JUnit version 4.13.2
.02:07:07.772 [DEBUG - main] (RandomUtils.java:49) Using random seed: 1779070027763
02:07:07.840 [INFO - main] (RetryRule.java:138) Creating RetryStatement with result reporter and retry count of 0 (no retries)
02:07:07.912 [INFO - main] (KuduTestHarness.java:155) Creating a new MiniKuduCluster...
02:07:07.927 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskTHXZc_/test-tmp
02:07:07.932 [INFO - main] (KuduBinaryLocator.java:83) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/release/bin
02:07:07.935 [INFO - main] (MiniKuduCluster.java:217) Starting process: [/tmp/dist-test-taskTHXZc_/java/build/dist-test/../../../build/release/bin/kudu, test, mini_cluster, --serialization=pb]
02:07:08.130 [DEBUG - main] (MiniKuduCluster.java:185) Request: create_cluster {
  num_masters: 3
  num_tservers: 3
  enable_kerberos: false
  cluster_root: "/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017"
  hms_mode: NONE
  mini_kdc_options {
  }
  principal: "kudu"
  mini_oidc_options {
  }
}

02:07:08.143 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
02:07:08.145 [DEBUG - main] (MiniKuduCluster.java:185) Request: start_cluster {
}

02:07:08.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.146346 22988 mini_oidc.cc:150] Starting JWKS server
02:07:08.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.147110 22988 webserver.cc:533] Webserver started at http://127.0.0.1:44893/ using document root <none> and password file <none>
02:07:08.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.147203 22988 mini_oidc.cc:186] Starting OIDC Discovery server
02:07:08.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.147367 22988 webserver.cc:533] Webserver started at http://127.0.0.1:39257/ using document root <none> and password file <none>
02:07:08.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-05-18T02:07:08Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
02:07:08.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-05-18T02:07:08Z Disabled control of system clock
02:07:08.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.159538 22988 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskTHXZc_/build/release/bin/kudu
02:07:08.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskTHXZc_/build/release/bin/kudu
02:07:08.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-0/wal
02:07:08.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-0/data
02:07:08.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:07:08.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:07:08.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:07:08.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:08.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:08.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:08.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-0/logs
02:07:08.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-0/data/info.pb
02:07:08.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:08.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:07:08.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:07:08.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:07:08.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:07:08.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:08.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:08.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:08.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
02:07:08.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:07:08.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
02:07:08.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
02:07:08.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.115.62:38055
02:07:08.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.115.62
02:07:08.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:07:08.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.115.20:33307
02:07:08.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:08.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:08.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:07:08.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
02:07:08.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.22.115.62:38055,127.22.115.61:46503,127.22.115.60:37831
02:07:08.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
02:07:08.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:44893/jwks.json with env {}
02:07:08.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.240530 22999 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:07:08.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.240706 22999 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:07:08.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.240727 22999 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:07:08.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.242600 22999 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
02:07:08.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.242668 22999 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:07:08.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.242690 22999 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
02:07:08.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.242708 22999 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
02:07:08.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.242726 22999 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
02:07:08.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.242741 22999 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:44893/jwks.json
02:07:08.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.245042 22999 master_runner.cc:387] Master server non-default flags:
02:07:08.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:08.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.115.20:33307
02:07:08.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:08.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:07:08.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-0/data
02:07:08.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-0/wal
02:07:08.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
02:07:08.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.22.115.62:38055,127.22.115.61:46503,127.22.115.60:37831
02:07:08.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:08.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:08.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
02:07:08.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.115.62:38055
02:07:08.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
02:07:08.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:07:08.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
02:07:08.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:44893/jwks.json
02:07:08.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:08.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:08.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-0/data/info.pb
02:07:08.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.115.62
02:07:08.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:07:08.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:07:08.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.22999
02:07:08.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:08.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:07:08.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:07:08.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:08.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-0/logs
02:07:08.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:08.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:07:08.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:07:08.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
02:07:08.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:07:08.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
02:07:08.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
02:07:08.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 01:43:15 UTC on e7f111948823
02:07:08.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11949
02:07:08.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.245352 22999 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:07:08.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.245599 22999 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:07:08.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.248956 23004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:08.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.249099 23005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:08.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.249119 22999 server_base.cc:1061] running on GCE node
02:07:08.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.249413 23007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:08.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.249658 22999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:07:08.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.249897 22999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:07:08.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.251039 22999 hybrid_clock.cc:648] HybridClock initialized: now 1779070028251016 us; error 39 us; skew 500 ppm
02:07:08.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.252359 22999 webserver.cc:533] Webserver started at http://127.22.115.62:44877/ using document root <none> and password file <none>
02:07:08.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.252562 22999 fs_manager.cc:362] Metadata directory not provided
02:07:08.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.252619 22999 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:07:08.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.252725 22999 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
02:07:08.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.253808 22999 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-0/data/instance:
02:07:08.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "6f5f6ff7d26b4a83946b092a263b34e6"
02:07:08.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:07:08 on dist-test-slave-37fl"
02:07:08.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.254182 22999 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-0/wal/instance:
02:07:08.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "6f5f6ff7d26b4a83946b092a263b34e6"
02:07:08.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:07:08 on dist-test-slave-37fl"
02:07:08.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.255782 22999 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.003s	sys 0.000s
02:07:08.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.256793 23013 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:07:08.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.257068 22999 fs_manager.cc:730] Time spent opening block manager: real 0.001s	user 0.001s	sys 0.000s
02:07:08.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.257120 22999 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-0/data,/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-0/wal
02:07:08.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "6f5f6ff7d26b4a83946b092a263b34e6"
02:07:08.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:07:08 on dist-test-slave-37fl"
02:07:08.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.257162 22999 fs_report.cc:389] FS layout report
02:07:08.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:07:08.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-0/wal
02:07:08.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-0/wal
02:07:08.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-0/data/data
02:07:08.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:07:08.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:07:08.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:07:08.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:07:08.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:07:08.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:07:08.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:07:08.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:07:08.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:07:08.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:07:08.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:07:08.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.274407 22999 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:07:08.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.274693 22999 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:07:08.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.274777 22999 kserver.cc:163] Server-wide thread pool size limit: 3276
02:07:08.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.278883 22999 rpc_server.cc:307] RPC server started. Bound to: 127.22.115.62:38055
02:07:08.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.278968 23065 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.115.62:38055 every 8 connection(s)
02:07:08.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.279186 22999 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-0/data/info.pb
02:07:08.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.279784 23066 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
02:07:08.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.281240 23066 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.115.62" port: 38055 } has no permanent_uuid. Determining permanent_uuid...
02:07:08.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.287415 23066 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.115.61" port: 46503 } has no permanent_uuid. Determining permanent_uuid...
02:07:08.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.287624 22988 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskTHXZc_/build/release/bin/kudu as pid 22999
02:07:08.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.287714 22988 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-0/wal/instance
02:07:08.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.288704 22988 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskTHXZc_/build/release/bin/kudu
02:07:08.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskTHXZc_/build/release/bin/kudu
02:07:08.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-1/wal
02:07:08.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-1/data
02:07:08.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:07:08.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:07:08.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:07:08.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:08.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:08.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:08.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-1/logs
02:07:08.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-1/data/info.pb
02:07:08.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:08.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:07:08.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:07:08.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:07:08.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:07:08.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:08.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:08.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:08.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
02:07:08.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:07:08.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
02:07:08.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
02:07:08.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.115.61:46503
02:07:08.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.115.61
02:07:08.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:07:08.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.115.20:33307
02:07:08.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:08.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:08.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:07:08.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
02:07:08.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.22.115.62:38055,127.22.115.61:46503,127.22.115.60:37831
02:07:08.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
02:07:08.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:44893/jwks.json with env {}
02:07:08.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.288844 23014 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.115.61:46503: connect: Connection refused (error 111)
02:07:08.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.292385 23066 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.22.115.61:46503: Network error: Client connection negotiation failed: client connection to 127.22.115.61:46503: connect: Connection refused (error 111)
02:07:08.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.341593 23066 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.115.61" port: 46503 } attempt: 1
02:07:08.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.342509 23066 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.22.115.61:46503: Network error: Client connection negotiation failed: client connection to 127.22.115.61:46503: connect: Connection refused (error 111)
02:07:08.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.368733 23071 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:07:08.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.368957 23071 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:07:08.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.368990 23071 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:07:08.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.370944 23071 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
02:07:08.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.371034 23071 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:07:08.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.371057 23071 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
02:07:08.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.371075 23071 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
02:07:08.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.371095 23071 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
02:07:08.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.371114 23071 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:44893/jwks.json
02:07:08.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.373257 23071 master_runner.cc:387] Master server non-default flags:
02:07:08.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:08.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.115.20:33307
02:07:08.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:08.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:07:08.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-1/data
02:07:08.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-1/wal
02:07:08.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
02:07:08.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.22.115.62:38055,127.22.115.61:46503,127.22.115.60:37831
02:07:08.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:08.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:08.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
02:07:08.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.115.61:46503
02:07:08.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
02:07:08.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:07:08.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
02:07:08.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:44893/jwks.json
02:07:08.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:08.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:08.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-1/data/info.pb
02:07:08.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.115.61
02:07:08.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:07:08.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:07:08.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.23071
02:07:08.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:08.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:07:08.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:07:08.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:08.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-1/logs
02:07:08.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:08.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:07:08.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:07:08.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
02:07:08.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:07:08.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
02:07:08.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
02:07:08.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 01:43:15 UTC on e7f111948823
02:07:08.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11949
02:07:08.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.373553 23071 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:07:08.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.373833 23071 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:07:08.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.376734 23076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:08.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.376788 23077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:08.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.377214 23079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:08.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.377522 23071 server_base.cc:1061] running on GCE node
02:07:08.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.377741 23071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:07:08.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.378000 23071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:07:08.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.379123 23071 hybrid_clock.cc:648] HybridClock initialized: now 1779070028379100 us; error 40 us; skew 500 ppm
02:07:08.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.380414 23071 webserver.cc:533] Webserver started at http://127.22.115.61:35829/ using document root <none> and password file <none>
02:07:08.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.380613 23071 fs_manager.cc:362] Metadata directory not provided
02:07:08.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.380666 23071 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:07:08.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.380766 23071 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
02:07:08.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.381961 23071 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-1/data/instance:
02:07:08.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "2c226a1a266e4d65a502c68f1dea1fbc"
02:07:08.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:07:08 on dist-test-slave-37fl"
02:07:08.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.382344 23071 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-1/wal/instance:
02:07:08.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "2c226a1a266e4d65a502c68f1dea1fbc"
02:07:08.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:07:08 on dist-test-slave-37fl"
02:07:08.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.384121 23071 fs_manager.cc:696] Time spent creating directory manager: real 0.002s	user 0.003s	sys 0.000s
02:07:08.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.384859 23085 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:07:08.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.384989 23071 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.001s	sys 0.000s
02:07:08.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.385038 23071 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-1/data,/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-1/wal
02:07:08.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "2c226a1a266e4d65a502c68f1dea1fbc"
02:07:08.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:07:08 on dist-test-slave-37fl"
02:07:08.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.385092 23071 fs_report.cc:389] FS layout report
02:07:08.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:07:08.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-1/wal
02:07:08.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-1/wal
02:07:08.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-1/data/data
02:07:08.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:07:08.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:07:08.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:07:08.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:07:08.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:07:08.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:07:08.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:07:08.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:07:08.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:07:08.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:07:08.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:07:08.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.403568 23071 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:07:08.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.403904 23071 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:07:08.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.404058 23071 kserver.cc:163] Server-wide thread pool size limit: 3276
02:07:08.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.409134 23071 rpc_server.cc:307] RPC server started. Bound to: 127.22.115.61:46503
02:07:08.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.409210 23137 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.115.61:46503 every 8 connection(s)
02:07:08.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.409548 23071 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-1/data/info.pb
02:07:08.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.410192 23138 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
02:07:08.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.410687 23066 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.115.61" port: 46503 } attempt: 2
02:07:08.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.412930 23138 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.115.62" port: 38055 } has no permanent_uuid. Determining permanent_uuid...
02:07:08.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.414834 22988 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskTHXZc_/build/release/bin/kudu as pid 23071
02:07:08.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.414928 22988 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-1/wal/instance
02:07:08.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.415321 22988 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskTHXZc_/build/release/bin/kudu
02:07:08.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskTHXZc_/build/release/bin/kudu
02:07:08.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-2/wal
02:07:08.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-2/data
02:07:08.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:07:08.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:07:08.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:07:08.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:08.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:08.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:08.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-2/logs
02:07:08.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-2/data/info.pb
02:07:08.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:08.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:07:08.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:07:08.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:07:08.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:07:08.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:08.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:08.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:08.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
02:07:08.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:07:08.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
02:07:08.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
02:07:08.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.115.60:37831
02:07:08.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.115.60
02:07:08.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:07:08.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.115.20:33307
02:07:08.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:08.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:08.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:07:08.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
02:07:08.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.22.115.62:38055,127.22.115.61:46503,127.22.115.60:37831
02:07:08.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
02:07:08.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:44893/jwks.json with env {}
02:07:08.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.417275 23066 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.115.60" port: 37831 } has no permanent_uuid. Determining permanent_uuid...
02:07:08.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.417816 23066 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.22.115.60:37831: Network error: Client connection negotiation failed: client connection to 127.22.115.60:37831: connect: Connection refused (error 111)
02:07:08.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.418156 23138 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.115.61" port: 46503 } has no permanent_uuid. Determining permanent_uuid...
02:07:08.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.424834 23138 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.115.60" port: 37831 } has no permanent_uuid. Determining permanent_uuid...
02:07:08.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.425200 23087 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.115.60:37831: connect: Connection refused (error 111)
02:07:08.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.427150 23138 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.22.115.60:37831: Network error: Client connection negotiation failed: client connection to 127.22.115.60:37831: connect: Connection refused (error 111)
02:07:08.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.460988 23066 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.115.60" port: 37831 } attempt: 1
02:07:08.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.462045 23066 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.22.115.60:37831: Network error: Client connection negotiation failed: client connection to 127.22.115.60:37831: connect: Connection refused (error 111)
02:07:08.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.476368 23138 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.115.60" port: 37831 } attempt: 1
02:07:08.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.477294 23138 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.22.115.60:37831: Network error: Client connection negotiation failed: client connection to 127.22.115.60:37831: connect: Connection refused (error 111)
02:07:08.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.499100 23142 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:07:08.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.499290 23142 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:07:08.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.499311 23142 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:07:08.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.500947 23142 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
02:07:08.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.501003 23142 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:07:08.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.501016 23142 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
02:07:08.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.501029 23142 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
02:07:08.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.501044 23142 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
02:07:08.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.501097 23142 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:44893/jwks.json
02:07:08.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.503073 23142 master_runner.cc:387] Master server non-default flags:
02:07:08.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:08.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.115.20:33307
02:07:08.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:08.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:07:08.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-2/data
02:07:08.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-2/wal
02:07:08.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
02:07:08.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.22.115.62:38055,127.22.115.61:46503,127.22.115.60:37831
02:07:08.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:08.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:08.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
02:07:08.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.115.60:37831
02:07:08.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
02:07:08.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:07:08.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
02:07:08.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:44893/jwks.json
02:07:08.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:08.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:08.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-2/data/info.pb
02:07:08.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.115.60
02:07:08.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:07:08.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:07:08.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.23142
02:07:08.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:08.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:07:08.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:07:08.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:08.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-2/logs
02:07:08.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:08.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:07:08.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:07:08.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
02:07:08.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:07:08.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
02:07:08.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
02:07:08.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 01:43:15 UTC on e7f111948823
02:07:08.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11949
02:07:08.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.503293 23142 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:07:08.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.503479 23142 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:07:08.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.506096 23149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:08.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.506850 23142 server_base.cc:1061] running on GCE node
02:07:08.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.506937 23151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:08.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.506953 23148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:08.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.507195 23142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:07:08.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.507513 23142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:07:08.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.508715 23142 hybrid_clock.cc:648] HybridClock initialized: now 1779070028508695 us; error 34 us; skew 500 ppm
02:07:08.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.509208 23066 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.115.60" port: 37831 } attempt: 2
02:07:08.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.509806 23066 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.22.115.60:37831: Network error: Client connection negotiation failed: client connection to 127.22.115.60:37831: connect: Connection refused (error 111)
02:07:08.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.510083 23142 webserver.cc:533] Webserver started at http://127.22.115.60:46301/ using document root <none> and password file <none>
02:07:08.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.510305 23142 fs_manager.cc:362] Metadata directory not provided
02:07:08.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.510362 23142 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:07:08.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.510466 23142 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
02:07:08.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.511626 23142 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-2/data/instance:
02:07:08.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b0ea07bc9aa94dc3b2ff3a298811fccf"
02:07:08.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:07:08 on dist-test-slave-37fl"
02:07:08.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.512084 23142 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-2/wal/instance:
02:07:08.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b0ea07bc9aa94dc3b2ff3a298811fccf"
02:07:08.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:07:08 on dist-test-slave-37fl"
02:07:08.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.513638 23142 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.001s	sys 0.000s
02:07:08.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.514612 23157 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:07:08.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.514854 23142 fs_manager.cc:730] Time spent opening block manager: real 0.001s	user 0.001s	sys 0.000s
02:07:08.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.514978 23142 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-2/data,/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-2/wal
02:07:08.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b0ea07bc9aa94dc3b2ff3a298811fccf"
02:07:08.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:07:08 on dist-test-slave-37fl"
02:07:08.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.515051 23142 fs_report.cc:389] FS layout report
02:07:08.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:07:08.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-2/wal
02:07:08.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-2/wal
02:07:08.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-2/data/data
02:07:08.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:07:08.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:07:08.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:07:08.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:07:08.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:07:08.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:07:08.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:07:08.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:07:08.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:07:08.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:07:08.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:07:08.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.535871 23142 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:07:08.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.536309 23142 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:07:08.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.536437 23142 kserver.cc:163] Server-wide thread pool size limit: 3276
02:07:08.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.540472 23142 rpc_server.cc:307] RPC server started. Bound to: 127.22.115.60:37831
02:07:08.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.540505 23209 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.115.60:37831 every 8 connection(s)
02:07:08.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.540889 23142 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-2/data/info.pb
02:07:08.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.541556 23210 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
02:07:08.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.541635 22988 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskTHXZc_/build/release/bin/kudu as pid 23142
02:07:08.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.541742 22988 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/master-2/wal/instance
02:07:08.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.544154 23210 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.115.62" port: 38055 } has no permanent_uuid. Determining permanent_uuid...
02:07:08.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.545461 23138 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.115.60" port: 37831 } attempt: 2
02:07:08.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.549861 23210 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.115.61" port: 46503 } has no permanent_uuid. Determining permanent_uuid...
02:07:08.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.551364 23138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c226a1a266e4d65a502c68f1dea1fbc: Bootstrap starting.
02:07:08.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.552338 23138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c226a1a266e4d65a502c68f1dea1fbc: Neither blocks nor log segments found. Creating new log.
02:07:08.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.553252 23210 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.115.60" port: 37831 } has no permanent_uuid. Determining permanent_uuid...
02:07:08.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.554337 23138 log.cc:826] T 00000000000000000000000000000000 P 2c226a1a266e4d65a502c68f1dea1fbc: Log is configured to *not* fsync() on all Append() calls
02:07:08.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.555415 23138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c226a1a266e4d65a502c68f1dea1fbc: No bootstrap required, opened a new log
02:07:08.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.556913 23210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0ea07bc9aa94dc3b2ff3a298811fccf: Bootstrap starting.
02:07:08.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.557092 23138 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2c226a1a266e4d65a502c68f1dea1fbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f5f6ff7d26b4a83946b092a263b34e6" member_type: VOTER last_known_addr { host: "127.22.115.62" port: 38055 } } peers { permanent_uuid: "2c226a1a266e4d65a502c68f1dea1fbc" member_type: VOTER last_known_addr { host: "127.22.115.61" port: 46503 } } peers { permanent_uuid: "b0ea07bc9aa94dc3b2ff3a298811fccf" member_type: VOTER last_known_addr { host: "127.22.115.60" port: 37831 } }
02:07:08.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.557225 23138 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2c226a1a266e4d65a502c68f1dea1fbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
02:07:08.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.557262 23138 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2c226a1a266e4d65a502c68f1dea1fbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c226a1a266e4d65a502c68f1dea1fbc, State: Initialized, Role: FOLLOWER
02:07:08.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.557374 23138 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c226a1a266e4d65a502c68f1dea1fbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f5f6ff7d26b4a83946b092a263b34e6" member_type: VOTER last_known_addr { host: "127.22.115.62" port: 38055 } } peers { permanent_uuid: "2c226a1a266e4d65a502c68f1dea1fbc" member_type: VOTER last_known_addr { host: "127.22.115.61" port: 46503 } } peers { permanent_uuid: "b0ea07bc9aa94dc3b2ff3a298811fccf" member_type: VOTER last_known_addr { host: "127.22.115.60" port: 37831 } }
02:07:08.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.557601 23210 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0ea07bc9aa94dc3b2ff3a298811fccf: Neither blocks nor log segments found. Creating new log.
02:07:08.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.557655 23138 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2c226a1a266e4d65a502c68f1dea1fbc [sys.catalog]: configured and running, proceeding with master startup.
02:07:08.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.557642 23219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c226a1a266e4d65a502c68f1dea1fbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f5f6ff7d26b4a83946b092a263b34e6" member_type: VOTER last_known_addr { host: "127.22.115.62" port: 38055 } } peers { permanent_uuid: "2c226a1a266e4d65a502c68f1dea1fbc" member_type: VOTER last_known_addr { host: "127.22.115.61" port: 46503 } } peers { permanent_uuid: "b0ea07bc9aa94dc3b2ff3a298811fccf" member_type: VOTER last_known_addr { host: "127.22.115.60" port: 37831 } } }
02:07:08.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.557756 23219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c226a1a266e4d65a502c68f1dea1fbc [sys.catalog]: This master's current role is: FOLLOWER
02:07:08.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.557868 23210 log.cc:826] T 00000000000000000000000000000000 P b0ea07bc9aa94dc3b2ff3a298811fccf: Log is configured to *not* fsync() on all Append() calls
02:07:08.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.559751 23231 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 2c226a1a266e4d65a502c68f1dea1fbc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
02:07:08.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.559805 23231 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
02:07:08.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.559831 23210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0ea07bc9aa94dc3b2ff3a298811fccf: No bootstrap required, opened a new log
02:07:08.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.560289 23071 master_runner.cc:187] Error getting master registration for 127.22.115.62:38055: OK, instance_id { permanent_uuid: "6f5f6ff7d26b4a83946b092a263b34e6" instance_seqno: 1779070028277770 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
02:07:08.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.562016 23071 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
02:07:08.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.562088 23210 raft_consensus.cc:359] T 00000000000000000000000000000000 P b0ea07bc9aa94dc3b2ff3a298811fccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f5f6ff7d26b4a83946b092a263b34e6" member_type: VOTER last_known_addr { host: "127.22.115.62" port: 38055 } } peers { permanent_uuid: "2c226a1a266e4d65a502c68f1dea1fbc" member_type: VOTER last_known_addr { host: "127.22.115.61" port: 46503 } } peers { permanent_uuid: "b0ea07bc9aa94dc3b2ff3a298811fccf" member_type: VOTER last_known_addr { host: "127.22.115.60" port: 37831 } }
02:07:08.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.562208 23210 raft_consensus.cc:385] T 00000000000000000000000000000000 P b0ea07bc9aa94dc3b2ff3a298811fccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
02:07:08.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.562243 23210 raft_consensus.cc:740] T 00000000000000000000000000000000 P b0ea07bc9aa94dc3b2ff3a298811fccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0ea07bc9aa94dc3b2ff3a298811fccf, State: Initialized, Role: FOLLOWER
02:07:08.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.562331 23210 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0ea07bc9aa94dc3b2ff3a298811fccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f5f6ff7d26b4a83946b092a263b34e6" member_type: VOTER last_known_addr { host: "127.22.115.62" port: 38055 } } peers { permanent_uuid: "2c226a1a266e4d65a502c68f1dea1fbc" member_type: VOTER last_known_addr { host: "127.22.115.61" port: 46503 } } peers { permanent_uuid: "b0ea07bc9aa94dc3b2ff3a298811fccf" member_type: VOTER last_known_addr { host: "127.22.115.60" port: 37831 } }
02:07:08.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.562633 23210 sys_catalog.cc:565] T 00000000000000000000000000000000 P b0ea07bc9aa94dc3b2ff3a298811fccf [sys.catalog]: configured and running, proceeding with master startup.
02:07:08.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.564155 23233 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0ea07bc9aa94dc3b2ff3a298811fccf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f5f6ff7d26b4a83946b092a263b34e6" member_type: VOTER last_known_addr { host: "127.22.115.62" port: 38055 } } peers { permanent_uuid: "2c226a1a266e4d65a502c68f1dea1fbc" member_type: VOTER last_known_addr { host: "127.22.115.61" port: 46503 } } peers { permanent_uuid: "b0ea07bc9aa94dc3b2ff3a298811fccf" member_type: VOTER last_known_addr { host: "127.22.115.60" port: 37831 } } }
02:07:08.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.564857 23244 catalog_manager.cc:1593] T 00000000000000000000000000000000 P b0ea07bc9aa94dc3b2ff3a298811fccf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
02:07:08.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.565512 23244 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
02:07:08.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.565563 23142 master_runner.cc:187] Error getting master registration for 127.22.115.62:38055: OK, instance_id { permanent_uuid: "6f5f6ff7d26b4a83946b092a263b34e6" instance_seqno: 1779070028277770 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
02:07:08.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.567356 23142 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
02:07:08.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.564919 23233 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0ea07bc9aa94dc3b2ff3a298811fccf [sys.catalog]: This master's current role is: FOLLOWER
02:07:08.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.616964 23066 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.115.60" port: 37831 } attempt: 3
02:07:08.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.621842 23066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f5f6ff7d26b4a83946b092a263b34e6: Bootstrap starting.
02:07:08.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.622646 23066 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f5f6ff7d26b4a83946b092a263b34e6: Neither blocks nor log segments found. Creating new log.
02:07:08.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.622936 23066 log.cc:826] T 00000000000000000000000000000000 P 6f5f6ff7d26b4a83946b092a263b34e6: Log is configured to *not* fsync() on all Append() calls
02:07:08.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.623903 23066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f5f6ff7d26b4a83946b092a263b34e6: No bootstrap required, opened a new log
02:07:08.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.625571 23066 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6f5f6ff7d26b4a83946b092a263b34e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f5f6ff7d26b4a83946b092a263b34e6" member_type: VOTER last_known_addr { host: "127.22.115.62" port: 38055 } } peers { permanent_uuid: "2c226a1a266e4d65a502c68f1dea1fbc" member_type: VOTER last_known_addr { host: "127.22.115.61" port: 46503 } } peers { permanent_uuid: "b0ea07bc9aa94dc3b2ff3a298811fccf" member_type: VOTER last_known_addr { host: "127.22.115.60" port: 37831 } }
02:07:08.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.625710 23066 raft_consensus.cc:385] T 00000000000000000000000000000000 P 6f5f6ff7d26b4a83946b092a263b34e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
02:07:08.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.625742 23066 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6f5f6ff7d26b4a83946b092a263b34e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f5f6ff7d26b4a83946b092a263b34e6, State: Initialized, Role: FOLLOWER
02:07:08.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.625838 23066 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f5f6ff7d26b4a83946b092a263b34e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f5f6ff7d26b4a83946b092a263b34e6" member_type: VOTER last_known_addr { host: "127.22.115.62" port: 38055 } } peers { permanent_uuid: "2c226a1a266e4d65a502c68f1dea1fbc" member_type: VOTER last_known_addr { host: "127.22.115.61" port: 46503 } } peers { permanent_uuid: "b0ea07bc9aa94dc3b2ff3a298811fccf" member_type: VOTER last_known_addr { host: "127.22.115.60" port: 37831 } }
02:07:08.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.626143 23066 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6f5f6ff7d26b4a83946b092a263b34e6 [sys.catalog]: configured and running, proceeding with master startup.
02:07:08.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.626693 23247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f5f6ff7d26b4a83946b092a263b34e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f5f6ff7d26b4a83946b092a263b34e6" member_type: VOTER last_known_addr { host: "127.22.115.62" port: 38055 } } peers { permanent_uuid: "2c226a1a266e4d65a502c68f1dea1fbc" member_type: VOTER last_known_addr { host: "127.22.115.61" port: 46503 } } peers { permanent_uuid: "b0ea07bc9aa94dc3b2ff3a298811fccf" member_type: VOTER last_known_addr { host: "127.22.115.60" port: 37831 } } }
02:07:08.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.627041 23247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f5f6ff7d26b4a83946b092a263b34e6 [sys.catalog]: This master's current role is: FOLLOWER
02:07:08.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.628042 23258 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 6f5f6ff7d26b4a83946b092a263b34e6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
02:07:08.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.628346 23258 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
02:07:08.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.629964 22999 master_runner.cc:264] No leader master found from master 6f5f6ff7d26b4a83946b092a263b34e6
02:07:08.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.630009 22999 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
02:07:08.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.654757 23091 tablet.cc:2404] T 00000000000000000000000000000000 P 2c226a1a266e4d65a502c68f1dea1fbc: Can't schedule compaction. Clean time has not been advanced past its initial value.
02:07:08.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.731405 23219 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2c226a1a266e4d65a502c68f1dea1fbc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
02:07:08.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.731547 23219 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2c226a1a266e4d65a502c68f1dea1fbc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f5f6ff7d26b4a83946b092a263b34e6" member_type: VOTER last_known_addr { host: "127.22.115.62" port: 38055 } } peers { permanent_uuid: "2c226a1a266e4d65a502c68f1dea1fbc" member_type: VOTER last_known_addr { host: "127.22.115.61" port: 46503 } } peers { permanent_uuid: "b0ea07bc9aa94dc3b2ff3a298811fccf" member_type: VOTER last_known_addr { host: "127.22.115.60" port: 37831 } }
02:07:08.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.731781 23219 leader_election.cc:290] T 00000000000000000000000000000000 P 2c226a1a266e4d65a502c68f1dea1fbc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f5f6ff7d26b4a83946b092a263b34e6 (127.22.115.62:38055), b0ea07bc9aa94dc3b2ff3a298811fccf (127.22.115.60:37831)
02:07:08.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.732221 23184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2c226a1a266e4d65a502c68f1dea1fbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0ea07bc9aa94dc3b2ff3a298811fccf" is_pre_election: true
02:07:08.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.732219 23040 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2c226a1a266e4d65a502c68f1dea1fbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f5f6ff7d26b4a83946b092a263b34e6" is_pre_election: true
02:07:08.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.732348 23040 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6f5f6ff7d26b4a83946b092a263b34e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c226a1a266e4d65a502c68f1dea1fbc in term 0.
02:07:08.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.732348 23184 raft_consensus.cc:2468] T 00000000000000000000000000000000 P b0ea07bc9aa94dc3b2ff3a298811fccf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c226a1a266e4d65a502c68f1dea1fbc in term 0.
02:07:08.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.732527 23087 leader_election.cc:304] T 00000000000000000000000000000000 P 2c226a1a266e4d65a502c68f1dea1fbc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c226a1a266e4d65a502c68f1dea1fbc, b0ea07bc9aa94dc3b2ff3a298811fccf; no voters: 
02:07:08.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.732653 23219 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2c226a1a266e4d65a502c68f1dea1fbc [term 0 FOLLOWER]: Leader pre-election won for term 1
02:07:08.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.732692 23219 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2c226a1a266e4d65a502c68f1dea1fbc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
02:07:08.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.732712 23219 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2c226a1a266e4d65a502c68f1dea1fbc [term 0 FOLLOWER]: Advancing to term 1
02:07:08.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.733594 23219 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2c226a1a266e4d65a502c68f1dea1fbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f5f6ff7d26b4a83946b092a263b34e6" member_type: VOTER last_known_addr { host: "127.22.115.62" port: 38055 } } peers { permanent_uuid: "2c226a1a266e4d65a502c68f1dea1fbc" member_type: VOTER last_known_addr { host: "127.22.115.61" port: 46503 } } peers { permanent_uuid: "b0ea07bc9aa94dc3b2ff3a298811fccf" member_type: VOTER last_known_addr { host: "127.22.115.60" port: 37831 } }
02:07:08.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.733718 23219 leader_election.cc:290] T 00000000000000000000000000000000 P 2c226a1a266e4d65a502c68f1dea1fbc [CANDIDATE]: Term 1 election: Requested vote from peers 6f5f6ff7d26b4a83946b092a263b34e6 (127.22.115.62:38055), b0ea07bc9aa94dc3b2ff3a298811fccf (127.22.115.60:37831)
02:07:08.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.733947 23184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2c226a1a266e4d65a502c68f1dea1fbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0ea07bc9aa94dc3b2ff3a298811fccf"
02:07:08.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.734019 23184 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b0ea07bc9aa94dc3b2ff3a298811fccf [term 0 FOLLOWER]: Advancing to term 1
02:07:08.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.734925 23184 raft_consensus.cc:2468] T 00000000000000000000000000000000 P b0ea07bc9aa94dc3b2ff3a298811fccf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c226a1a266e4d65a502c68f1dea1fbc in term 1.
02:07:08.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.735069 23040 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2c226a1a266e4d65a502c68f1dea1fbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f5f6ff7d26b4a83946b092a263b34e6"
02:07:08.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.735100 23087 leader_election.cc:304] T 00000000000000000000000000000000 P 2c226a1a266e4d65a502c68f1dea1fbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c226a1a266e4d65a502c68f1dea1fbc, b0ea07bc9aa94dc3b2ff3a298811fccf; no voters: 
02:07:08.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.735167 23219 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2c226a1a266e4d65a502c68f1dea1fbc [term 1 FOLLOWER]: Leader election won for term 1
02:07:08.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.735157 23040 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6f5f6ff7d26b4a83946b092a263b34e6 [term 0 FOLLOWER]: Advancing to term 1
02:07:08.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.735322 23219 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2c226a1a266e4d65a502c68f1dea1fbc [term 1 LEADER]: Becoming Leader. State: Replica: 2c226a1a266e4d65a502c68f1dea1fbc, State: Running, Role: LEADER
02:07:08.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.735405 23219 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c226a1a266e4d65a502c68f1dea1fbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f5f6ff7d26b4a83946b092a263b34e6" member_type: VOTER last_known_addr { host: "127.22.115.62" port: 38055 } } peers { permanent_uuid: "2c226a1a266e4d65a502c68f1dea1fbc" member_type: VOTER last_known_addr { host: "127.22.115.61" port: 46503 } } peers { permanent_uuid: "b0ea07bc9aa94dc3b2ff3a298811fccf" member_type: VOTER last_known_addr { host: "127.22.115.60" port: 37831 } }
02:07:08.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.735720 23219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c226a1a266e4d65a502c68f1dea1fbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c226a1a266e4d65a502c68f1dea1fbc. Latest consensus state: current_term: 1 leader_uuid: "2c226a1a266e4d65a502c68f1dea1fbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f5f6ff7d26b4a83946b092a263b34e6" member_type: VOTER last_known_addr { host: "127.22.115.62" port: 38055 } } peers { permanent_uuid: "2c226a1a266e4d65a502c68f1dea1fbc" member_type: VOTER last_known_addr { host: "127.22.115.61" port: 46503 } } peers { permanent_uuid: "b0ea07bc9aa94dc3b2ff3a298811fccf" member_type: VOTER last_known_addr { host: "127.22.115.60" port: 37831 } } }
02:07:08.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.735787 23219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c226a1a266e4d65a502c68f1dea1fbc [sys.catalog]: This master's current role is: LEADER
02:07:08.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.736023 23040 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6f5f6ff7d26b4a83946b092a263b34e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c226a1a266e4d65a502c68f1dea1fbc in term 1.
02:07:08.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.736465 23262 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
02:07:08.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.736661 23262 catalog_manager.cc:1519] Initializing Kudu cluster ID...
02:07:08.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.738482 23184 raft_consensus.cc:1275] T 00000000000000000000000000000000 P b0ea07bc9aa94dc3b2ff3a298811fccf [term 1 FOLLOWER]: Refusing update from remote peer 2c226a1a266e4d65a502c68f1dea1fbc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
02:07:08.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.738734 23040 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6f5f6ff7d26b4a83946b092a263b34e6 [term 1 FOLLOWER]: Refusing update from remote peer 2c226a1a266e4d65a502c68f1dea1fbc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
02:07:08.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.739073 23219 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 2c226a1a266e4d65a502c68f1dea1fbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f5f6ff7d26b4a83946b092a263b34e6" member_type: VOTER last_known_addr { host: "127.22.115.62" port: 38055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
02:07:08.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.739214 23219 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 2c226a1a266e4d65a502c68f1dea1fbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0ea07bc9aa94dc3b2ff3a298811fccf" member_type: VOTER last_known_addr { host: "127.22.115.60" port: 37831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
02:07:08.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.741114 23233 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0ea07bc9aa94dc3b2ff3a298811fccf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c226a1a266e4d65a502c68f1dea1fbc. Latest consensus state: current_term: 1 leader_uuid: "2c226a1a266e4d65a502c68f1dea1fbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f5f6ff7d26b4a83946b092a263b34e6" member_type: VOTER last_known_addr { host: "127.22.115.62" port: 38055 } } peers { permanent_uuid: "2c226a1a266e4d65a502c68f1dea1fbc" member_type: VOTER last_known_addr { host: "127.22.115.61" port: 46503 } } peers { permanent_uuid: "b0ea07bc9aa94dc3b2ff3a298811fccf" member_type: VOTER last_known_addr { host: "127.22.115.60" port: 37831 } } }
02:07:08.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.741235 23233 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0ea07bc9aa94dc3b2ff3a298811fccf [sys.catalog]: This master's current role is: FOLLOWER
02:07:08.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.741605 23263 mvcc.cc:204] Tried to move back new op lower bound from 7287070837710090240 to 7287070837700644864. Current Snapshot: MvccSnapshot[applied={T|T < 7287070837710090240}]
02:07:08.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.741907 23219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c226a1a266e4d65a502c68f1dea1fbc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2c226a1a266e4d65a502c68f1dea1fbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f5f6ff7d26b4a83946b092a263b34e6" member_type: VOTER last_known_addr { host: "127.22.115.62" port: 38055 } } peers { permanent_uuid: "2c226a1a266e4d65a502c68f1dea1fbc" member_type: VOTER last_known_addr { host: "127.22.115.61" port: 46503 } } peers { permanent_uuid: "b0ea07bc9aa94dc3b2ff3a298811fccf" member_type: VOTER last_known_addr { host: "127.22.115.60" port: 37831 } } }
02:07:08.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.741978 23219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c226a1a266e4d65a502c68f1dea1fbc [sys.catalog]: This master's current role is: LEADER
02:07:08.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.743377 23219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c226a1a266e4d65a502c68f1dea1fbc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2c226a1a266e4d65a502c68f1dea1fbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f5f6ff7d26b4a83946b092a263b34e6" member_type: VOTER last_known_addr { host: "127.22.115.62" port: 38055 } } peers { permanent_uuid: "2c226a1a266e4d65a502c68f1dea1fbc" member_type: VOTER last_known_addr { host: "127.22.115.61" port: 46503 } } peers { permanent_uuid: "b0ea07bc9aa94dc3b2ff3a298811fccf" member_type: VOTER last_known_addr { host: "127.22.115.60" port: 37831 } } }
02:07:08.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.743446 23219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c226a1a266e4d65a502c68f1dea1fbc [sys.catalog]: This master's current role is: LEADER
02:07:08.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.743853 23233 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0ea07bc9aa94dc3b2ff3a298811fccf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2c226a1a266e4d65a502c68f1dea1fbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f5f6ff7d26b4a83946b092a263b34e6" member_type: VOTER last_known_addr { host: "127.22.115.62" port: 38055 } } peers { permanent_uuid: "2c226a1a266e4d65a502c68f1dea1fbc" member_type: VOTER last_known_addr { host: "127.22.115.61" port: 46503 } } peers { permanent_uuid: "b0ea07bc9aa94dc3b2ff3a298811fccf" member_type: VOTER last_known_addr { host: "127.22.115.60" port: 37831 } } }
02:07:08.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.743937 23233 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0ea07bc9aa94dc3b2ff3a298811fccf [sys.catalog]: This master's current role is: FOLLOWER
02:07:08.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.744225 23247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f5f6ff7d26b4a83946b092a263b34e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c226a1a266e4d65a502c68f1dea1fbc. Latest consensus state: current_term: 1 leader_uuid: "2c226a1a266e4d65a502c68f1dea1fbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f5f6ff7d26b4a83946b092a263b34e6" member_type: VOTER last_known_addr { host: "127.22.115.62" port: 38055 } } peers { permanent_uuid: "2c226a1a266e4d65a502c68f1dea1fbc" member_type: VOTER last_known_addr { host: "127.22.115.61" port: 46503 } } peers { permanent_uuid: "b0ea07bc9aa94dc3b2ff3a298811fccf" member_type: VOTER last_known_addr { host: "127.22.115.60" port: 37831 } } }
02:07:08.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.744305 23247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f5f6ff7d26b4a83946b092a263b34e6 [sys.catalog]: This master's current role is: FOLLOWER
02:07:08.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.744323 23264 mvcc.cc:204] Tried to move back new op lower bound from 7287070837710090240 to 7287070837700644864. Current Snapshot: MvccSnapshot[applied={T|T < 7287070837710090240}]
02:07:08.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.744390 23247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f5f6ff7d26b4a83946b092a263b34e6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2c226a1a266e4d65a502c68f1dea1fbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f5f6ff7d26b4a83946b092a263b34e6" member_type: VOTER last_known_addr { host: "127.22.115.62" port: 38055 } } peers { permanent_uuid: "2c226a1a266e4d65a502c68f1dea1fbc" member_type: VOTER last_known_addr { host: "127.22.115.61" port: 46503 } } peers { permanent_uuid: "b0ea07bc9aa94dc3b2ff3a298811fccf" member_type: VOTER last_known_addr { host: "127.22.115.60" port: 37831 } } }
02:07:08.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.744426 23247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f5f6ff7d26b4a83946b092a263b34e6 [sys.catalog]: This master's current role is: FOLLOWER
02:07:08.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.744424 23266 mvcc.cc:204] Tried to move back new op lower bound from 7287070837710090240 to 7287070837700644864. Current Snapshot: MvccSnapshot[applied={T|T < 7287070837710090240}]
02:07:08.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.745546 23262 catalog_manager.cc:1382] Generated new cluster ID: 27289adbe8114954b500a7f3a4efe510
02:07:08.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.745594 23262 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
02:07:08.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.766326 23262 catalog_manager.cc:1405] Generated new certificate authority record
02:07:08.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.768007 23262 catalog_manager.cc:1539] Loading token signing keys...
02:07:08.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.777737 23262 catalog_manager.cc:6069] T 00000000000000000000000000000000 P 2c226a1a266e4d65a502c68f1dea1fbc: Generated new TSK 0
02:07:08.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.777930 23262 catalog_manager.cc:1549] Initializing in-progress tserver states...
02:07:08.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.781386 22988 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskTHXZc_/build/release/bin/kudu
02:07:08.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskTHXZc_/build/release/bin/kudu
02:07:08.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/wal
02:07:08.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/data
02:07:08.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:07:08.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:07:08.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:07:08.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:08.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:08.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:08.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/logs
02:07:08.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/data/info.pb
02:07:08.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:08.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:07:08.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:07:08.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:07:08.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:07:08.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:08.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:08.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:08.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:07:08.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:07:08.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.115.1:0
02:07:08.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.115.1
02:07:08.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.115.1
02:07:08.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:07:08.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.115.62:38055,127.22.115.61:46503,127.22.115.60:37831
02:07:08.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.115.20:33307
02:07:08.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:08.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:08.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:07:08.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.868167 23273 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:07:08.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.868342 23273 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:07:08.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.868363 23273 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:07:08.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.869812 23273 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:07:08.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.869863 23273 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.115.1
02:07:08.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.871383 23273 tablet_server_runner.cc:78] Tablet server non-default flags:
02:07:08.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:08.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.115.20:33307
02:07:08.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:08.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:07:08.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/data
02:07:08.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/wal
02:07:08.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:08.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:08.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.115.1:0
02:07:08.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:07:08.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:08.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:08.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/data/info.pb
02:07:08.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.115.1
02:07:08.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:07:08.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.115.62:38055,127.22.115.61:46503,127.22.115.60:37831
02:07:08.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:07:08.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.23273
02:07:08.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:08.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:07:08.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:07:08.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:08.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.115.1
02:07:08.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/logs
02:07:08.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:08.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:07:08.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:07:08.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:07:08.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:07:08.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
02:07:08.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
02:07:08.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 01:43:15 UTC on e7f111948823
02:07:08.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11949
02:07:08.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.871578 23273 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:07:08.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.871760 23273 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:07:08.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.874238 23278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:08.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.874253 23281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:08.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:08.874428 23279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:08.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.874640 23273 server_base.cc:1061] running on GCE node
02:07:08.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.874788 23273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:07:08.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.875010 23273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:07:08.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.876161 23273 hybrid_clock.cc:648] HybridClock initialized: now 1779070028876128 us; error 45 us; skew 500 ppm
02:07:08.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.877529 23273 webserver.cc:533] Webserver started at http://127.22.115.1:35073/ using document root <none> and password file <none>
02:07:08.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.877741 23273 fs_manager.cc:362] Metadata directory not provided
02:07:08.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.877859 23273 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:07:08.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.878034 23273 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
02:07:08.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.879182 23273 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/data/instance:
02:07:08.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "619c08b8f3e246edabc0e4e315cf9201"
02:07:08.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:07:08 on dist-test-slave-37fl"
02:07:08.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.879551 23273 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/wal/instance:
02:07:08.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "619c08b8f3e246edabc0e4e315cf9201"
02:07:08.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:07:08 on dist-test-slave-37fl"
02:07:08.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.881027 23273 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.000s	sys 0.003s
02:07:08.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.881901 23287 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:07:08.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.882472 23273 fs_manager.cc:730] Time spent opening block manager: real 0.001s	user 0.000s	sys 0.001s
02:07:08.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.882550 23273 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/data,/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/wal
02:07:08.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "619c08b8f3e246edabc0e4e315cf9201"
02:07:08.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:07:08 on dist-test-slave-37fl"
02:07:08.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.882603 23273 fs_report.cc:389] FS layout report
02:07:08.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:07:08.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/wal
02:07:08.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/wal
02:07:08.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/data/data
02:07:08.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:07:08.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:07:08.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:07:08.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:07:08.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:07:08.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:07:08.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:07:08.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:07:08.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:07:08.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:07:08.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:07:08.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.910730 23273 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:07:08.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.911033 23273 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:07:08.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.911123 23273 kserver.cc:163] Server-wide thread pool size limit: 3276
02:07:08.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.911301 23273 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:07:08.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.911651 23273 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
02:07:08.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.911693 23273 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
02:07:08.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.911729 23273 ts_tablet_manager.cc:616] Registered 0 tablets
02:07:08.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.911751 23273 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
02:07:08.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.919158 23273 rpc_server.cc:307] RPC server started. Bound to: 127.22.115.1:46047
02:07:08.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.919260 23400 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.115.1:46047 every 8 connection(s)
02:07:08.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.919582 23273 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/data/info.pb
02:07:08.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.925103 23402 heartbeater.cc:344] Connected to a master server at 127.22.115.60:37831
02:07:08.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.925153 23403 heartbeater.cc:344] Connected to a master server at 127.22.115.61:46503
02:07:08.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.925225 23403 heartbeater.cc:461] Registering TS with master...
02:07:08.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.925225 23402 heartbeater.cc:461] Registering TS with master...
02:07:08.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.925402 23403 heartbeater.cc:507] Master 127.22.115.61:46503 requested a full tablet report, sending...
02:07:08.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.925405 23402 heartbeater.cc:507] Master 127.22.115.60:37831 requested a full tablet report, sending...
02:07:08.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.925841 23102 ts_manager.cc:194] Registered new tserver with Master: 619c08b8f3e246edabc0e4e315cf9201 (127.22.115.1:46047)
02:07:08.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.925899 23174 ts_manager.cc:194] Registered new tserver with Master: 619c08b8f3e246edabc0e4e315cf9201 (127.22.115.1:46047)
02:07:08.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.925946 23401 heartbeater.cc:344] Connected to a master server at 127.22.115.62:38055
02:07:08.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.925997 23401 heartbeater.cc:461] Registering TS with master...
02:07:08.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.926090 23401 heartbeater.cc:507] Master 127.22.115.62:38055 requested a full tablet report, sending...
02:07:08.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.926427 23030 ts_manager.cc:194] Registered new tserver with Master: 619c08b8f3e246edabc0e4e315cf9201 (127.22.115.1:46047)
02:07:08.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.926708 23102 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.115.1:45945
02:07:08.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.928040 22988 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskTHXZc_/build/release/bin/kudu as pid 23273
02:07:08.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.928154 22988 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/wal/instance
02:07:08.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:08.929368 22988 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskTHXZc_/build/release/bin/kudu
02:07:08.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskTHXZc_/build/release/bin/kudu
02:07:08.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/wal
02:07:08.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/data
02:07:08.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:07:08.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:07:08.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:07:08.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:08.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:08.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:08.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/logs
02:07:08.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/data/info.pb
02:07:08.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:08.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:07:08.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:07:08.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:07:08.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:07:08.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:08.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:08.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:08.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:07:08.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:07:08.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.115.2:0
02:07:08.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.115.2
02:07:08.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.115.2
02:07:08.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:07:08.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.115.62:38055,127.22.115.61:46503,127.22.115.60:37831
02:07:08.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.115.20:33307
02:07:08.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:08.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:08.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:07:09.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:09.011584 23408 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:07:09.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:09.011754 23408 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:07:09.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:09.011775 23408 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:07:09.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:09.013304 23408 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:07:09.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:09.013357 23408 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.115.2
02:07:09.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.014878 23408 tablet_server_runner.cc:78] Tablet server non-default flags:
02:07:09.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:09.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.115.20:33307
02:07:09.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:09.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:07:09.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/data
02:07:09.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/wal
02:07:09.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:09.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:09.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.115.2:0
02:07:09.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:07:09.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:09.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:09.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/data/info.pb
02:07:09.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.115.2
02:07:09.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:07:09.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.115.62:38055,127.22.115.61:46503,127.22.115.60:37831
02:07:09.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:07:09.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.23408
02:07:09.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:09.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:07:09.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:07:09.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:09.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.115.2
02:07:09.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/logs
02:07:09.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:09.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:07:09.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:07:09.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:07:09.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:07:09.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
02:07:09.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
02:07:09.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 01:43:15 UTC on e7f111948823
02:07:09.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11949
02:07:09.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.015058 23408 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:07:09.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.015237 23408 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:07:09.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:09.017912 23413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:09.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:09.017961 23414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:09.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:09.017997 23416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:09.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.018436 23408 server_base.cc:1061] running on GCE node
02:07:09.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.018570 23408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:07:09.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.018764 23408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:07:09.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.020022 23408 hybrid_clock.cc:648] HybridClock initialized: now 1779070029020000 us; error 35 us; skew 500 ppm
02:07:09.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.021035 23408 webserver.cc:533] Webserver started at http://127.22.115.2:37975/ using document root <none> and password file <none>
02:07:09.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.021188 23408 fs_manager.cc:362] Metadata directory not provided
02:07:09.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.021227 23408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:07:09.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.021299 23408 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
02:07:09.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.022101 23408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/data/instance:
02:07:09.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "85635564f2c94c458f506c870a826b34"
02:07:09.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:07:09 on dist-test-slave-37fl"
02:07:09.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.022363 23408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/wal/instance:
02:07:09.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "85635564f2c94c458f506c870a826b34"
02:07:09.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:07:09 on dist-test-slave-37fl"
02:07:09.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.023568 23408 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.002s	sys 0.000s
02:07:09.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.024335 23422 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:07:09.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.024554 23408 fs_manager.cc:730] Time spent opening block manager: real 0.001s	user 0.001s	sys 0.000s
02:07:09.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.024606 23408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/data,/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/wal
02:07:09.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "85635564f2c94c458f506c870a826b34"
02:07:09.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:07:09 on dist-test-slave-37fl"
02:07:09.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.024657 23408 fs_report.cc:389] FS layout report
02:07:09.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:07:09.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/wal
02:07:09.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/wal
02:07:09.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/data/data
02:07:09.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:07:09.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:07:09.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:07:09.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:07:09.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:07:09.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:07:09.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:07:09.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:07:09.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:07:09.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:07:09.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:07:09.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.057323 23408 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:07:09.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.057600 23408 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:07:09.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.057720 23408 kserver.cc:163] Server-wide thread pool size limit: 3276
02:07:09.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.057906 23408 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:07:09.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.058305 23408 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
02:07:09.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.058341 23408 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
02:07:09.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.058372 23408 ts_tablet_manager.cc:616] Registered 0 tablets
02:07:09.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.058391 23408 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
02:07:09.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.065464 23408 rpc_server.cc:307] RPC server started. Bound to: 127.22.115.2:43855
02:07:09.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.065544 23535 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.115.2:43855 every 8 connection(s)
02:07:09.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.065804 23408 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/data/info.pb
02:07:09.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.071206 23537 heartbeater.cc:344] Connected to a master server at 127.22.115.60:37831
02:07:09.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.071566 23538 heartbeater.cc:344] Connected to a master server at 127.22.115.61:46503
02:07:09.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.071486 23536 heartbeater.cc:344] Connected to a master server at 127.22.115.62:38055
02:07:09.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.071655 23536 heartbeater.cc:461] Registering TS with master...
02:07:09.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.071651 23537 heartbeater.cc:461] Registering TS with master...
02:07:09.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.071650 23538 heartbeater.cc:461] Registering TS with master...
02:07:09.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.071900 23537 heartbeater.cc:507] Master 127.22.115.60:37831 requested a full tablet report, sending...
02:07:09.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.071900 23536 heartbeater.cc:507] Master 127.22.115.62:38055 requested a full tablet report, sending...
02:07:09.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.071900 23538 heartbeater.cc:507] Master 127.22.115.61:46503 requested a full tablet report, sending...
02:07:09.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.072324 23174 ts_manager.cc:194] Registered new tserver with Master: 85635564f2c94c458f506c870a826b34 (127.22.115.2:43855)
02:07:09.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.072355 23030 ts_manager.cc:194] Registered new tserver with Master: 85635564f2c94c458f506c870a826b34 (127.22.115.2:43855)
02:07:09.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.072357 23102 ts_manager.cc:194] Registered new tserver with Master: 85635564f2c94c458f506c870a826b34 (127.22.115.2:43855)
02:07:09.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.072868 23102 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.115.2:49689
02:07:09.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.074698 22988 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskTHXZc_/build/release/bin/kudu as pid 23408
02:07:09.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.074767 22988 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/wal/instance
02:07:09.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.075875 22988 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskTHXZc_/build/release/bin/kudu
02:07:09.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskTHXZc_/build/release/bin/kudu
02:07:09.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/wal
02:07:09.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/data
02:07:09.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:07:09.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:07:09.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:07:09.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:09.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:09.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:09.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/logs
02:07:09.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/data/info.pb
02:07:09.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:09.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:07:09.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:07:09.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:07:09.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:07:09.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:09.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:09.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:09.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:07:09.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:07:09.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.115.3:0
02:07:09.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.115.3
02:07:09.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.115.3
02:07:09.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:07:09.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.115.62:38055,127.22.115.61:46503,127.22.115.60:37831
02:07:09.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.115.20:33307
02:07:09.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:09.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:09.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:07:09.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:09.152019 23543 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:07:09.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:09.152187 23543 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:07:09.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:09.152207 23543 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:07:09.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:09.153863 23543 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:07:09.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:09.153915 23543 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.115.3
02:07:09.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.155355 23543 tablet_server_runner.cc:78] Tablet server non-default flags:
02:07:09.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:09.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.115.20:33307
02:07:09.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:09.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:07:09.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/data
02:07:09.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/wal
02:07:09.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:09.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:09.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.115.3:0
02:07:09.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:07:09.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:09.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:09.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/data/info.pb
02:07:09.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.115.3
02:07:09.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:07:09.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.115.62:38055,127.22.115.61:46503,127.22.115.60:37831
02:07:09.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:07:09.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.23543
02:07:09.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:09.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:07:09.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:07:09.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:09.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.115.3
02:07:09.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/logs
02:07:09.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:09.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:07:09.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:07:09.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:07:09.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:07:09.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
02:07:09.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
02:07:09.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 01:43:15 UTC on e7f111948823
02:07:09.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11949
02:07:09.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.155534 23543 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:07:09.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.155691 23543 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:07:09.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:09.158232 23551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:09.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:09.158437 23549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:09.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:09.158546 23548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:09.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.158857 23543 server_base.cc:1061] running on GCE node
02:07:09.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.159027 23543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:07:09.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.159236 23543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:07:09.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.160354 23543 hybrid_clock.cc:648] HybridClock initialized: now 1779070029160331 us; error 34 us; skew 500 ppm
02:07:09.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.161324 23543 webserver.cc:533] Webserver started at http://127.22.115.3:45263/ using document root <none> and password file <none>
02:07:09.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.161466 23543 fs_manager.cc:362] Metadata directory not provided
02:07:09.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.161499 23543 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:07:09.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.161569 23543 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
02:07:09.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.162366 23543 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/data/instance:
02:07:09.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "174330aa0d3e49c69cab90b20e4f7a56"
02:07:09.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:07:09 on dist-test-slave-37fl"
02:07:09.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.162621 23543 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/wal/instance:
02:07:09.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "174330aa0d3e49c69cab90b20e4f7a56"
02:07:09.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:07:09 on dist-test-slave-37fl"
02:07:09.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.164048 23543 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.002s	sys 0.000s
02:07:09.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.164861 23557 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:07:09.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.165266 23543 fs_manager.cc:730] Time spent opening block manager: real 0.001s	user 0.001s	sys 0.000s
02:07:09.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.165342 23543 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/data,/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/wal
02:07:09.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "174330aa0d3e49c69cab90b20e4f7a56"
02:07:09.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:07:09 on dist-test-slave-37fl"
02:07:09.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.165397 23543 fs_report.cc:389] FS layout report
02:07:09.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:07:09.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/wal
02:07:09.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/wal
02:07:09.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/data/data
02:07:09.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:07:09.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:07:09.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:07:09.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:07:09.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:07:09.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:07:09.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:07:09.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:07:09.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:07:09.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:07:09.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:07:09.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.178642 23543 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:07:09.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.178892 23543 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:07:09.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.178982 23543 kserver.cc:163] Server-wide thread pool size limit: 3276
02:07:09.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.179136 23543 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:07:09.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.179380 23543 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
02:07:09.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.179407 23543 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
02:07:09.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.179428 23543 ts_tablet_manager.cc:616] Registered 0 tablets
02:07:09.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.179441 23543 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
02:07:09.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.185793 23543 rpc_server.cc:307] RPC server started. Bound to: 127.22.115.3:43855
02:07:09.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.185868 23670 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.115.3:43855 every 8 connection(s)
02:07:09.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.186220 23543 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/data/info.pb
02:07:09.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.190521 22988 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskTHXZc_/build/release/bin/kudu as pid 23543
02:07:09.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.190632 22988 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/wal/instance
02:07:09.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.192371 23671 heartbeater.cc:344] Connected to a master server at 127.22.115.62:38055
02:07:09.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.192466 23673 heartbeater.cc:344] Connected to a master server at 127.22.115.61:46503
02:07:09.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.192579 23672 heartbeater.cc:344] Connected to a master server at 127.22.115.60:37831
02:07:09.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.192773 23672 heartbeater.cc:461] Registering TS with master...
02:07:09.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.192735 23673 heartbeater.cc:461] Registering TS with master...
02:07:09.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.193137 23673 heartbeater.cc:507] Master 127.22.115.61:46503 requested a full tablet report, sending...
02:07:09.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.193138 23672 heartbeater.cc:507] Master 127.22.115.60:37831 requested a full tablet report, sending...
02:07:09.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.192735 23671 heartbeater.cc:461] Registering TS with master...
02:07:09.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.193578 23101 ts_manager.cc:194] Registered new tserver with Master: 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855)
02:07:09.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.193650 23174 ts_manager.cc:194] Registered new tserver with Master: 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855)
02:07:09.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.193774 23671 heartbeater.cc:507] Master 127.22.115.62:38055 requested a full tablet report, sending...
02:07:09.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.194090 23030 ts_manager.cc:194] Registered new tserver with Master: 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855)
02:07:09.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.194147 23101 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.115.3:54303
02:07:09.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.206220 22988 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
02:07:09.207 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
02:07:09.208 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_masters {
}

02:07:09.216 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_masters {
  masters {
    id {
      type: MASTER
      index: 0
    }
    bound_rpc_address {
      host: "127.22.115.62"
      port: 38055
    }
    bound_http_address {
      host: "127.22.115.62"
      port: 44877
    }
  }
  masters {
    id {
      type: MASTER
      index: 1
    }
    bound_rpc_address {
      host: "127.22.115.61"
      port: 46503
    }
    bound_http_address {
      host: "127.22.115.61"
      port: 35829
    }
  }
  masters {
    id {
      type: MASTER
      index: 2
    }
    bound_rpc_address {
      host: "127.22.115.60"
      port: 37831
    }
    bound_http_address {
      host: "127.22.115.60"
      port: 46301
    }
  }
}

02:07:09.219 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_tservers {
}

02:07:09.220 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_tservers {
  tservers {
    id {
      type: TSERVER
      index: 0
    }
    bound_rpc_address {
      host: "127.22.115.1"
      port: 46047
    }
    bound_http_address {
      host: "127.22.115.1"
      port: 35073
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 1
    }
    bound_rpc_address {
      host: "127.22.115.2"
      port: 43855
    }
    bound_http_address {
      host: "127.22.115.2"
      port: 37975
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 2
    }
    bound_rpc_address {
      host: "127.22.115.3"
      port: 43855
    }
    bound_http_address {
      host: "127.22.115.3"
      port: 45263
    }
  }
}

02:07:09.221 [INFO - main] (KuduTestHarness.java:157) Creating a new Kudu client...
02:07:09.396 [DEBUG - main] (CompositeStrategy.java:59) Unable to load modern Subject methods, relying only on legacy methods
java.lang.NoSuchMethodException: javax.security.auth.Subject.current()
	at java.base/java.lang.Class.getDeclaredMethod(Class.java:2677)
	at org.apache.kudu.client.internals.ModernStrategy.<init>(ModernStrategy.java:51)
	at org.apache.kudu.client.internals.CompositeStrategy.<init>(CompositeStrategy.java:52)
	at org.apache.kudu.client.internals.CompositeStrategy.<clinit>(CompositeStrategy.java:41)
	at org.apache.kudu.client.internals.SecurityManagerCompatibility.get(SecurityManagerCompatibility.java:43)
	at org.apache.kudu.client.SecurityContext.setupSubject(SecurityContext.java:155)
	at org.apache.kudu.client.SecurityContext.<init>(SecurityContext.java:138)
	at org.apache.kudu.client.AsyncKuduClient.<init>(AsyncKuduClient.java:431)
	at org.apache.kudu.client.AsyncKuduClient$AsyncKuduClientBuilder.build(AsyncKuduClient.java:3109)
	at org.apache.kudu.test.KuduTestHarness.before(KuduTestHarness.java:161)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:50)
	at org.apache.kudu.test.junit.RetryRule$RetryStatement.doOneAttemptAndReport(RetryRule.java:184)
	at org.apache.kudu.test.junit.RetryRule$RetryStatement.evaluate(RetryRule.java:244)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:27)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
	at org.junit.runner.JUnitCore.runMain(JUnitCore.java:77)
	at org.junit.runner.JUnitCore.main(JUnitCore.java:36)
02:07:09.445 [DEBUG - main] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 
02:07:09.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.566236 23244 catalog_manager.cc:1294] Loaded cluster ID: 27289adbe8114954b500a7f3a4efe510
02:07:09.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.566329 23244 catalog_manager.cc:1587] T 00000000000000000000000000000000 P b0ea07bc9aa94dc3b2ff3a298811fccf: loading cluster ID for follower catalog manager: success
02:07:09.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.567646 23244 catalog_manager.cc:1609] T 00000000000000000000000000000000 P b0ea07bc9aa94dc3b2ff3a298811fccf: acquiring CA information for follower catalog manager: success
02:07:09.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.568033 23244 catalog_manager.cc:1637] T 00000000000000000000000000000000 P b0ea07bc9aa94dc3b2ff3a298811fccf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
02:07:09.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.628983 23258 catalog_manager.cc:1294] Loaded cluster ID: 27289adbe8114954b500a7f3a4efe510
02:07:09.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.629067 23258 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 6f5f6ff7d26b4a83946b092a263b34e6: loading cluster ID for follower catalog manager: success
02:07:09.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.630412 23258 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 6f5f6ff7d26b4a83946b092a263b34e6: acquiring CA information for follower catalog manager: success
02:07:09.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.630710 23258 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 6f5f6ff7d26b4a83946b092a263b34e6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
02:07:09.632 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.22.115.60:37831(127.22.115.60:37831)] connecting to peer
02:07:09.691 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.22.115.62:38055(127.22.115.62:38055)] connecting to peer
02:07:09.692 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.22.115.61:46503(127.22.115.61:46503)] connecting to peer
02:07:09.709 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.22.115.60:37831(127.22.115.60:37831)] Successfully connected to peer
02:07:09.709 [DEBUG - kudu-nio-2] (Connection.java:757) [peer master-127.22.115.61:46503(127.22.115.61:46503)] Successfully connected to peer
02:07:09.709 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.22.115.62:38055(127.22.115.62:38055)] Successfully connected to peer
02:07:09.811 [DEBUG - kudu-nio-1] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.22.115.62
02:07:09.811 [DEBUG - kudu-nio-2] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.22.115.61
02:07:09.811 [DEBUG - kudu-nio-0] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.22.115.60
02:07:09.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:09.927407 23403 heartbeater.cc:499] Master 127.22.115.61:46503 was elected leader, sending a full tablet report...
02:07:10.048 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x74183abd, L:/127.0.0.1:42274 - R:/127.22.115.60:37831] using SASL/PLAIN
02:07:10.048 [DEBUG - kudu-nio-2] (Negotiator.java:919) Authenticated connection [id: 0x9b97a85d, L:/127.0.0.1:35506 - R:/127.22.115.61:46503] using SASL/PLAIN
02:07:10.048 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0xb426dd59, L:/127.0.0.1:59940 - R:/127.22.115.62:38055] using SASL/PLAIN
02:07:10.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:10.066156 23531 messenger.cc:194] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.22.115.2:43855
02:07:10.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.073628 23538 heartbeater.cc:499] Master 127.22.115.61:46503 was elected leader, sending a full tablet report...
02:07:10.083 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet Kudu Master for table 'Kudu Master' with partition [<start>, <end>)
02:07:10.084 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=CreateTable, tablet=null, attempt=1, TimeoutTracker(timeout=50000, elapsed=500), Traces: [0ms] refreshing cache from master, [28ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.115.60:37831, [97ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.115.62:38055, [99ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.115.61:46503, [466ms] Sub RPC ConnectToMaster: received response from server master-127.22.115.62:38055: OK, [466ms] Sub RPC ConnectToMaster: received response from server master-127.22.115.60:37831: OK, [467ms] Sub RPC ConnectToMaster: received response from server master-127.22.115.61:46503: OK, deferred=null) after lookup
02:07:10.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.130082 23101 catalog_manager.cc:2282] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35506:
02:07:10.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) name: "org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563"
02:07:10.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) schema {
02:07:10.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
02:07:10.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "key"
02:07:10.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
02:07:10.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: true
02:07:10.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
02:07:10.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
02:07:10.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
02:07:10.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
02:07:10.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
02:07:10.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
02:07:10.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column1_i"
02:07:10.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
02:07:10.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
02:07:10.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
02:07:10.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
02:07:10.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
02:07:10.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
02:07:10.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
02:07:10.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
02:07:10.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column2_i"
02:07:10.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
02:07:10.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
02:07:10.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
02:07:10.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
02:07:10.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
02:07:10.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
02:07:10.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
02:07:10.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
02:07:10.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column3_s"
02:07:10.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: STRING
02:07:10.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
02:07:10.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: true
02:07:10.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     encoding: DICT_ENCODING
02:07:10.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     compression: LZ4
02:07:10.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 4096
02:07:10.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
02:07:10.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
02:07:10.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
02:07:10.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
02:07:10.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column4_b"
02:07:10.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: BOOL
02:07:10.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
02:07:10.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
02:07:10.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
02:07:10.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
02:07:10.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
02:07:10.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
02:07:10.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
02:07:10.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) partition_schema {
02:07:10.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   range_schema {
02:07:10.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     columns {
02:07:10.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)       name: "key"
02:07:10.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     }
02:07:10.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
02:07:10.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
02:07:10.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:10.130445 23101 catalog_manager.cc:7058] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
02:07:10.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.142544 23335 tablet_service.cc:1511] Processing CreateTablet for tablet 76ab850ff70d4ccca2590ffea11986ed (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563 [id=cf7ed0a8d17d441d8e0b073aea9657d1]), partition=RANGE (key) PARTITION UNBOUNDED
02:07:10.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.142647 23470 tablet_service.cc:1511] Processing CreateTablet for tablet 76ab850ff70d4ccca2590ffea11986ed (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563 [id=cf7ed0a8d17d441d8e0b073aea9657d1]), partition=RANGE (key) PARTITION UNBOUNDED
02:07:10.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.142856 23335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76ab850ff70d4ccca2590ffea11986ed. 1 dirs total, 0 dirs full, 0 dirs failed
02:07:10.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.142855 23470 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76ab850ff70d4ccca2590ffea11986ed. 1 dirs total, 0 dirs full, 0 dirs failed
02:07:10.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.145620 23705 tablet_bootstrap.cc:492] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201: Bootstrap starting.
02:07:10.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.146272 23705 tablet_bootstrap.cc:654] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201: Neither blocks nor log segments found. Creating new log.
02:07:10.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.146585 23705 log.cc:826] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201: Log is configured to *not* fsync() on all Append() calls
02:07:10.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.146881 23706 tablet_bootstrap.cc:492] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34: Bootstrap starting.
02:07:10.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.147691 23706 tablet_bootstrap.cc:654] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34: Neither blocks nor log segments found. Creating new log.
02:07:10.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.148079 23706 log.cc:826] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34: Log is configured to *not* fsync() on all Append() calls
02:07:10.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.148518 23705 tablet_bootstrap.cc:492] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201: No bootstrap required, opened a new log
02:07:10.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.148607 23705 ts_tablet_manager.cc:1403] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201: Time spent bootstrapping tablet: real 0.003s	user 0.001s	sys 0.000s
02:07:10.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.149991 23705 raft_consensus.cc:359] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:10.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.150132 23705 raft_consensus.cc:385] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
02:07:10.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.150167 23705 raft_consensus.cc:740] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 619c08b8f3e246edabc0e4e315cf9201, State: Initialized, Role: FOLLOWER
02:07:10.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.150269 23705 consensus_queue.cc:260] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:10.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.150509 23705 ts_tablet_manager.cc:1434] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
02:07:10.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.152887 23706 tablet_bootstrap.cc:492] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34: No bootstrap required, opened a new log
02:07:10.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.152973 23706 ts_tablet_manager.cc:1403] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34: Time spent bootstrapping tablet: real 0.006s	user 0.001s	sys 0.000s
02:07:10.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.154170 23706 raft_consensus.cc:359] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:10.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.154311 23706 raft_consensus.cc:385] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
02:07:10.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.154345 23706 raft_consensus.cc:740] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85635564f2c94c458f506c870a826b34, State: Initialized, Role: FOLLOWER
02:07:10.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.154448 23706 consensus_queue.cc:260] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:10.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.154651 23706 ts_tablet_manager.cc:1434] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
02:07:10.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.157469 23605 tablet_service.cc:1511] Processing CreateTablet for tablet 76ab850ff70d4ccca2590ffea11986ed (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563 [id=cf7ed0a8d17d441d8e0b073aea9657d1]), partition=RANGE (key) PARTITION UNBOUNDED
02:07:10.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.157738 23605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76ab850ff70d4ccca2590ffea11986ed. 1 dirs total, 0 dirs full, 0 dirs failed
02:07:10.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.159866 23711 tablet_bootstrap.cc:492] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56: Bootstrap starting.
02:07:10.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.160609 23711 tablet_bootstrap.cc:654] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56: Neither blocks nor log segments found. Creating new log.
02:07:10.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.160905 23711 log.cc:826] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56: Log is configured to *not* fsync() on all Append() calls
02:07:10.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.161486 23711 tablet_bootstrap.cc:492] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56: No bootstrap required, opened a new log
02:07:10.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.161566 23711 ts_tablet_manager.cc:1403] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56: Time spent bootstrapping tablet: real 0.002s	user 0.001s	sys 0.000s
02:07:10.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.162664 23711 raft_consensus.cc:359] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:10.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.162796 23711 raft_consensus.cc:385] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
02:07:10.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.162830 23711 raft_consensus.cc:740] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 174330aa0d3e49c69cab90b20e4f7a56, State: Initialized, Role: FOLLOWER
02:07:10.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.162930 23711 consensus_queue.cc:260] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:10.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.163151 23711 ts_tablet_manager.cc:1434] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
02:07:10.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.163477 23673 heartbeater.cc:499] Master 127.22.115.61:46503 was elected leader, sending a full tablet report...
02:07:10.164 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:894) Opened table cf7ed0a8d17d441d8e0b073aea9657d1
02:07:10.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:10.170913 23404 tablet.cc:2404] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201: Can't schedule compaction. Clean time has not been advanced past its initial value.
02:07:10.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:10.187515 23674 tablet.cc:2404] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56: Can't schedule compaction. Clean time has not been advanced past its initial value.
02:07:10.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:10.187556 23666 messenger.cc:194] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.22.115.3:43855
02:07:10.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:10.317549 23539 tablet.cc:2404] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34: Can't schedule compaction. Clean time has not been advanced past its initial value.
02:07:10.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.374267 23713 raft_consensus.cc:493] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
02:07:10.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.374442 23713 raft_consensus.cc:515] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:10.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.374788 23713 leader_election.cc:290] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 619c08b8f3e246edabc0e4e315cf9201 (127.22.115.1:46047), 85635564f2c94c458f506c870a826b34 (127.22.115.2:43855)
02:07:10.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.378798 23490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76ab850ff70d4ccca2590ffea11986ed" candidate_uuid: "174330aa0d3e49c69cab90b20e4f7a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85635564f2c94c458f506c870a826b34" is_pre_election: true
02:07:10.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.378798 23355 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76ab850ff70d4ccca2590ffea11986ed" candidate_uuid: "174330aa0d3e49c69cab90b20e4f7a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "619c08b8f3e246edabc0e4e315cf9201" is_pre_election: true
02:07:10.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.378944 23355 raft_consensus.cc:2468] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 174330aa0d3e49c69cab90b20e4f7a56 in term 0.
02:07:10.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.378999 23490 raft_consensus.cc:2468] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 174330aa0d3e49c69cab90b20e4f7a56 in term 0.
02:07:10.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.379191 23561 leader_election.cc:304] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 174330aa0d3e49c69cab90b20e4f7a56, 619c08b8f3e246edabc0e4e315cf9201; no voters: 
02:07:10.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.379333 23713 raft_consensus.cc:2804] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 0 FOLLOWER]: Leader pre-election won for term 1
02:07:10.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.379395 23713 raft_consensus.cc:493] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
02:07:10.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.379423 23713 raft_consensus.cc:3060] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 0 FOLLOWER]: Advancing to term 1
02:07:10.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.380228 23713 raft_consensus.cc:515] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:10.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.380416 23713 leader_election.cc:290] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [CANDIDATE]: Term 1 election: Requested vote from peers 619c08b8f3e246edabc0e4e315cf9201 (127.22.115.1:46047), 85635564f2c94c458f506c870a826b34 (127.22.115.2:43855)
02:07:10.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.380645 23355 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76ab850ff70d4ccca2590ffea11986ed" candidate_uuid: "174330aa0d3e49c69cab90b20e4f7a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "619c08b8f3e246edabc0e4e315cf9201"
02:07:10.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.380662 23490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76ab850ff70d4ccca2590ffea11986ed" candidate_uuid: "174330aa0d3e49c69cab90b20e4f7a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85635564f2c94c458f506c870a826b34"
02:07:10.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.380749 23490 raft_consensus.cc:3060] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 0 FOLLOWER]: Advancing to term 1
02:07:10.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.380751 23355 raft_consensus.cc:3060] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 0 FOLLOWER]: Advancing to term 1
02:07:10.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.381491 23355 raft_consensus.cc:2468] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 174330aa0d3e49c69cab90b20e4f7a56 in term 1.
02:07:10.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.381621 23490 raft_consensus.cc:2468] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 174330aa0d3e49c69cab90b20e4f7a56 in term 1.
02:07:10.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.381701 23561 leader_election.cc:304] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 174330aa0d3e49c69cab90b20e4f7a56, 619c08b8f3e246edabc0e4e315cf9201; no voters: 
02:07:10.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.381788 23713 raft_consensus.cc:2804] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 1 FOLLOWER]: Leader election won for term 1
02:07:10.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.381956 23713 raft_consensus.cc:697] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 1 LEADER]: Becoming Leader. State: Replica: 174330aa0d3e49c69cab90b20e4f7a56, State: Running, Role: LEADER
02:07:10.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.382054 23713 consensus_queue.cc:237] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:10.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.382793 23101 catalog_manager.cc:5696] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 reported cstate change: term changed from 0 to 1, leader changed from <none> to 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3). New cstate: current_term: 1 leader_uuid: "174330aa0d3e49c69cab90b20e4f7a56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } health_report { overall_health: HEALTHY } } }
02:07:10.454 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:894) Opened table cf7ed0a8d17d441d8e0b073aea9657d1
02:07:10.469 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:10.481 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299988, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)[L],619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047),85635564f2c94c458f506c870a826b34(127.22.115.2:43855)]}]
02:07:10.482 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=null, attempt=1, sequence_id=-1, TimeoutTracker(timeout=30000, elapsed=21), Traces: [0ms] refreshing cache from master, [1ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [7ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=null) after lookup
02:07:10.491 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] connecting to peer
02:07:10.494 [DEBUG - kudu-nio-3] (Connection.java:757) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] Successfully connected to peer
02:07:10.548 [DEBUG - kudu-nio-3] (Negotiator.java:919) Authenticated connection [id: 0xeaf519cb, L:/127.0.0.1:40516 - R:/127.22.115.3:43855] using TOKEN/null
02:07:10.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.551460 23490 raft_consensus.cc:1275] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 1 FOLLOWER]: Refusing update from remote peer 174330aa0d3e49c69cab90b20e4f7a56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
02:07:10.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.551517 23355 raft_consensus.cc:1275] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 1 FOLLOWER]: Refusing update from remote peer 174330aa0d3e49c69cab90b20e4f7a56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
02:07:10.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.551819 23713 consensus_queue.cc:1048] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
02:07:10.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.552281 23713 consensus_queue.cc:1048] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
02:07:10.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.554804 23720 mvcc.cc:204] Tried to move back new op lower bound from 7287070845135577088 to 7287070844445368320. Current Snapshot: MvccSnapshot[applied={T|T < 7287070845135577088}]
02:07:10.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.555027 23723 mvcc.cc:204] Tried to move back new op lower bound from 7287070845135577088 to 7287070844445368320. Current Snapshot: MvccSnapshot[applied={T|T < 7287070845135577088}]
02:07:10.628 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 3 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563, tablet=76ab850ff70d4ccca2590ffea11986ed, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:07:10.630 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.22.115.3:43855
02:07:10.631 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

02:07:10.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:10.632100 22988 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskTHXZc_/build/release/bin/kudu with pid 23543
02:07:10.638 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:07:10.641 [DEBUG - kudu-nio-3] (Connection.java:239) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] handling channelInactive
02:07:10.642 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] cleaning up while in state READY due to: connection closed
02:07:10.642 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855) for tablet 76ab850ff70d4ccca2590ffea11986ed: connection closed
02:07:10.652 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:10.652 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)[L],619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047),85635564f2c94c458f506c870a826b34(127.22.115.2:43855)]}]
02:07:10.652 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=2, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=13), Traces: [0ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [3ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [10ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [12ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@1175099903(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@6bc1e83 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@44934c17 -> wakeup thread Time-limited test)) after lookup
02:07:10.653 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] connecting to peer
02:07:10.656 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:10.656 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] cleaning up while in state CONNECTING due to: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:10.656 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:10.673 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:10.673 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)[L],619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047),85635564f2c94c458f506c870a826b34(127.22.115.2:43855)]}]
02:07:10.673 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=4, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=34), Traces: [0ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [3ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [10ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [12ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [14ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [17ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [17ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [33ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@1175099903(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@6bc1e83 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1132417316 after retry RPC@1997100714), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@44934c17 -> wakeup thread Time-limited test -> (continuation of Deferred@1132417316 after retry RPC@1997100714))) after lookup
02:07:10.674 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] connecting to peer
02:07:10.676 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:10.676 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] cleaning up while in state CONNECTING due to: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:10.677 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:10.713 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:10.713 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)[L],619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047),85635564f2c94c458f506c870a826b34(127.22.115.2:43855)]}]
02:07:10.714 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=6, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=74), Traces: [0ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [3ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [10ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [12ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [14ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [17ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [17ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [33ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [34ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [37ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [38ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [71ms] refreshing cache from master, [71ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [73ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@1175099903(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@6bc1e83 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1132417316 after retry RPC@1997100714) -> (continuation of Deferred@1735581545 after retry RPC@1516425819), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@44934c17 -> wakeup thread Time-limited test -> (continuation of Deferred@1132417316 after retry RPC@1997100714) -> (continuation of Deferred@1735581545 after retry RPC@1516425819))) after lookup
02:07:10.714 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] connecting to peer
02:07:10.716 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:10.716 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] cleaning up while in state CONNECTING due to: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:10.717 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:10.753 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:10.753 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)[L],619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047),85635564f2c94c458f506c870a826b34(127.22.115.2:43855)]}]
02:07:10.754 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=8, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=114), Traces: [0ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [3ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [10ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [12ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [14ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [17ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [17ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [33ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [34ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [37ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [38ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [71ms] refreshing cache from master, [71ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [73ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [75ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [78ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [78ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [110ms] refreshing cache from master, [111ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [113ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@1175099903(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@6bc1e83 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1132417316 after retry RPC@1997100714) -> (continuation of Deferred@1735581545 after retry RPC@1516425819) -> (continuation of Deferred@586524000 after retry RPC@1196136272), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@44934c17 -> wakeup thread Time-limited test -> (continuation of Deferred@1132417316 after retry RPC@1997100714) -> (continuation of Deferred@1735581545 after retry RPC@1516425819) -> (continuation of Deferred@586524000 after retry RPC@1196136272))) after lookup
02:07:10.755 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] connecting to peer
02:07:10.756 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:10.757 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] cleaning up while in state CONNECTING due to: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:10.757 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:11.093 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:11.093 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)[L],619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047),85635564f2c94c458f506c870a826b34(127.22.115.2:43855)]}]
02:07:11.093 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=10, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=454), Traces: [0ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [3ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [10ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [12ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [14ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [17ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [17ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [33ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [34ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [37ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [38ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [71ms] refreshing cache from master, [71ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [73ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [75ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [78ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [78ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [110ms] refreshing cache from master, [111ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [113ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [115ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [118ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [118ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [451ms] refreshing cache from master, [451ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [453ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@1175099903(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@6bc1e83 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1132417316 after retry RPC@1997100714) -> (continuation of Deferred@1735581545 after retry RPC@1516425819) -> (continuation of Deferred@586524000 after retry RPC@1196136272) -> (continuation of Deferred@1649529942 after retry RPC@812061348), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@44934c17 -> wakeup thread Time-limited test -> (continuation of Deferred@1132417316 after retry RPC@1997100714) -> (continuation of Deferred@1735581545 after retry RPC@1516425819) -> (continuation of Deferred@586524000 after retry RPC@1196136272) -> (continuation of Deferred@1649529942 after retry RPC@812061348))) after lookup
02:07:11.094 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] connecting to peer
02:07:11.095 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:11.095 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] cleaning up while in state CONNECTING due to: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:11.096 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:12.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:12.085201 23733 raft_consensus.cc:493] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 174330aa0d3e49c69cab90b20e4f7a56)
02:07:12.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:12.085254 23734 raft_consensus.cc:493] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 174330aa0d3e49c69cab90b20e4f7a56)
02:07:12.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:12.085310 23733 raft_consensus.cc:515] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:12.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:12.085368 23734 raft_consensus.cc:515] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:12.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:12.085613 23733 leader_election.cc:290] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85635564f2c94c458f506c870a826b34 (127.22.115.2:43855), 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855)
02:07:12.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:12.086206 23288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.115.3:43855: connect: Connection refused (error 111)
02:07:12.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:12.086838 23288 leader_election.cc:336] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855): Network error: Client connection negotiation failed: client connection to 127.22.115.3:43855: connect: Connection refused (error 111)
02:07:12.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:12.087095 23734 leader_election.cc:290] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 619c08b8f3e246edabc0e4e315cf9201 (127.22.115.1:46047), 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855)
02:07:12.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:12.087245 23423 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.115.3:43855: connect: Connection refused (error 111)
02:07:12.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:12.087826 23423 leader_election.cc:336] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855): Network error: Client connection negotiation failed: client connection to 127.22.115.3:43855: connect: Connection refused (error 111)
02:07:12.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:12.089186 23490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76ab850ff70d4ccca2590ffea11986ed" candidate_uuid: "619c08b8f3e246edabc0e4e315cf9201" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "85635564f2c94c458f506c870a826b34" is_pre_election: true
02:07:12.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:12.089330 23490 raft_consensus.cc:2468] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 619c08b8f3e246edabc0e4e315cf9201 in term 1.
02:07:12.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:12.089449 23355 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76ab850ff70d4ccca2590ffea11986ed" candidate_uuid: "85635564f2c94c458f506c870a826b34" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "619c08b8f3e246edabc0e4e315cf9201" is_pre_election: true
02:07:12.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:12.089541 23355 raft_consensus.cc:2468] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85635564f2c94c458f506c870a826b34 in term 1.
02:07:12.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:12.089561 23291 leader_election.cc:304] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 619c08b8f3e246edabc0e4e315cf9201, 85635564f2c94c458f506c870a826b34; no voters: 174330aa0d3e49c69cab90b20e4f7a56
02:07:12.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:12.089699 23733 raft_consensus.cc:2804] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 1 FOLLOWER]: Leader pre-election won for term 2
02:07:12.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:12.089744 23733 raft_consensus.cc:493] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 174330aa0d3e49c69cab90b20e4f7a56)
02:07:12.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:12.089735 23426 leader_election.cc:304] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 619c08b8f3e246edabc0e4e315cf9201, 85635564f2c94c458f506c870a826b34; no voters: 174330aa0d3e49c69cab90b20e4f7a56
02:07:12.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:12.089771 23733 raft_consensus.cc:3060] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 1 FOLLOWER]: Advancing to term 2
02:07:12.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:12.089844 23734 raft_consensus.cc:2804] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 1 FOLLOWER]: Leader pre-election won for term 2
02:07:12.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:12.089881 23734 raft_consensus.cc:493] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 174330aa0d3e49c69cab90b20e4f7a56)
02:07:12.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:12.089906 23734 raft_consensus.cc:3060] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 1 FOLLOWER]: Advancing to term 2
02:07:12.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:12.090721 23734 raft_consensus.cc:515] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:12.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:12.090714 23733 raft_consensus.cc:515] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:12.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:12.090880 23733 leader_election.cc:290] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [CANDIDATE]: Term 2 election: Requested vote from peers 85635564f2c94c458f506c870a826b34 (127.22.115.2:43855), 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855)
02:07:12.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:12.090932 23734 leader_election.cc:290] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [CANDIDATE]: Term 2 election: Requested vote from peers 619c08b8f3e246edabc0e4e315cf9201 (127.22.115.1:46047), 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855)
02:07:12.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:12.091228 23490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76ab850ff70d4ccca2590ffea11986ed" candidate_uuid: "619c08b8f3e246edabc0e4e315cf9201" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "85635564f2c94c458f506c870a826b34"
02:07:12.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:12.091234 23355 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76ab850ff70d4ccca2590ffea11986ed" candidate_uuid: "85635564f2c94c458f506c870a826b34" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "619c08b8f3e246edabc0e4e315cf9201"
02:07:12.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:12.091310 23490 raft_consensus.cc:2393] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 619c08b8f3e246edabc0e4e315cf9201 in current term 2: Already voted for candidate 85635564f2c94c458f506c870a826b34 in this term.
02:07:12.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:12.091347 23355 raft_consensus.cc:2393] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 85635564f2c94c458f506c870a826b34 in current term 2: Already voted for candidate 619c08b8f3e246edabc0e4e315cf9201 in this term.
02:07:12.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:12.091917 23423 leader_election.cc:336] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855): Network error: Client connection negotiation failed: client connection to 127.22.115.3:43855: connect: Connection refused (error 111)
02:07:12.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:12.092005 23423 leader_election.cc:304] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85635564f2c94c458f506c870a826b34; no voters: 174330aa0d3e49c69cab90b20e4f7a56, 619c08b8f3e246edabc0e4e315cf9201
02:07:12.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:12.092051 23288 leader_election.cc:336] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855): Network error: Client connection negotiation failed: client connection to 127.22.115.3:43855: connect: Connection refused (error 111)
02:07:12.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:12.092074 23734 raft_consensus.cc:2749] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
02:07:12.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:12.092096 23288 leader_election.cc:304] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 619c08b8f3e246edabc0e4e315cf9201; no voters: 174330aa0d3e49c69cab90b20e4f7a56, 85635564f2c94c458f506c870a826b34
02:07:12.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:12.092206 23733 raft_consensus.cc:2749] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
02:07:12.693 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:12.693 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)[L],619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047),85635564f2c94c458f506c870a826b34(127.22.115.2:43855)]}]
02:07:12.694 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=12, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=2054), Traces: [0ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [3ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [10ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [12ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [14ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [17ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [17ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [33ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [34ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [37ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [38ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [71ms] refreshing cache from master, [71ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [73ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [75ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [78ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [78ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [110ms] refreshing cache from master, [111ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [113ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [115ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [118ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [118ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [451ms] refreshing cache from master, [451ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [453ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [455ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [456ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [457ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [2050ms] refreshing cache from master, [2051ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [2053ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@1175099903(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@6bc1e83 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1132417316 after retry RPC@1997100714) -> (continuation of Deferred@1735581545 after retry RPC@1516425819) -> (continuation of Deferred@586524000 after retry RPC@1196136272) -> (continuation of Deferred@1649529942 after retry RPC@812061348) -> (continuation of Deferred@1476746041 after retry RPC@1358921903), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@44934c17 -> wakeup thread Time-limited test -> (continuation of Deferred@1132417316 after retry RPC@1997100714) -> (continuation of Deferred@1735581545 after retry RPC@1516425819) -> (continuation of Deferred@586524000 after retry RPC@1196136272) -> (continuation of Deferred@1649529942 after retry RPC@812061348) -> (continuation of Deferred@1476746041 after retry RPC@1358921903))) after lookup
02:07:12.695 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] connecting to peer
02:07:12.696 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:12.696 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] cleaning up while in state CONNECTING due to: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:12.696 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:13.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:13.740736 23744 raft_consensus.cc:493] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
02:07:13.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:13.740841 23744 raft_consensus.cc:515] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:13.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:13.741025 23744 leader_election.cc:290] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 619c08b8f3e246edabc0e4e315cf9201 (127.22.115.1:46047), 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855)
02:07:13.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:13.741319 23355 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76ab850ff70d4ccca2590ffea11986ed" candidate_uuid: "85635564f2c94c458f506c870a826b34" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "619c08b8f3e246edabc0e4e315cf9201" is_pre_election: true
02:07:13.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:13.741418 23355 raft_consensus.cc:2468] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85635564f2c94c458f506c870a826b34 in term 2.
02:07:13.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:13.741595 23426 leader_election.cc:304] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 619c08b8f3e246edabc0e4e315cf9201, 85635564f2c94c458f506c870a826b34; no voters: 
02:07:13.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:13.741699 23744 raft_consensus.cc:2804] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 2 FOLLOWER]: Leader pre-election won for term 3
02:07:13.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:13.741739 23744 raft_consensus.cc:493] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
02:07:13.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:13.741765 23744 raft_consensus.cc:3060] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 2 FOLLOWER]: Advancing to term 3
02:07:13.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:13.741959 23423 leader_election.cc:336] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855): Network error: Client connection negotiation failed: client connection to 127.22.115.3:43855: connect: Connection refused (error 111)
02:07:13.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:13.742638 23744 raft_consensus.cc:515] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:13.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:13.742822 23744 leader_election.cc:290] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [CANDIDATE]: Term 3 election: Requested vote from peers 619c08b8f3e246edabc0e4e315cf9201 (127.22.115.1:46047), 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855)
02:07:13.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:13.743014 23355 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76ab850ff70d4ccca2590ffea11986ed" candidate_uuid: "85635564f2c94c458f506c870a826b34" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "619c08b8f3e246edabc0e4e315cf9201"
02:07:13.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:13.743093 23355 raft_consensus.cc:3060] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 2 FOLLOWER]: Advancing to term 3
02:07:13.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:13.743471 23423 leader_election.cc:336] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855): Network error: Client connection negotiation failed: client connection to 127.22.115.3:43855: connect: Connection refused (error 111)
02:07:13.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:13.743775 23355 raft_consensus.cc:2468] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85635564f2c94c458f506c870a826b34 in term 3.
02:07:13.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:13.743947 23426 leader_election.cc:304] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 619c08b8f3e246edabc0e4e315cf9201, 85635564f2c94c458f506c870a826b34; no voters: 174330aa0d3e49c69cab90b20e4f7a56
02:07:13.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:13.744081 23744 raft_consensus.cc:2804] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 3 FOLLOWER]: Leader election won for term 3
02:07:13.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:13.744175 23744 raft_consensus.cc:697] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 3 LEADER]: Becoming Leader. State: Replica: 85635564f2c94c458f506c870a826b34, State: Running, Role: LEADER
02:07:13.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:13.744244 23744 consensus_queue.cc:237] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:13.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:13.744954 23101 catalog_manager.cc:5696] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 reported cstate change: term changed from 1 to 3, leader changed from 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3) to 85635564f2c94c458f506c870a826b34 (127.22.115.2). New cstate: current_term: 3 leader_uuid: "85635564f2c94c458f506c870a826b34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } health_report { overall_health: UNKNOWN } } }
02:07:14.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:14.232115 23423 consensus_peers.cc:597] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 -> Peer 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855): Couldn't send request to peer 174330aa0d3e49c69cab90b20e4f7a56. Status: Network error: Client connection negotiation failed: client connection to 127.22.115.3:43855: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:07:14.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:14.323838 23355 raft_consensus.cc:1275] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 3 FOLLOWER]: Refusing update from remote peer 85635564f2c94c458f506c870a826b34: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
02:07:14.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:14.324223 23747 consensus_queue.cc:1048] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
02:07:15.193 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:15.193 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855),619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047),85635564f2c94c458f506c870a826b34(127.22.115.2:43855)[L]]}]
02:07:15.194 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=14, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=4554), Traces: [0ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [3ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [10ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [12ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [14ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [17ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [17ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [33ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [34ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [37ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [38ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [71ms] refreshing cache from master, [71ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [73ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [75ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [78ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [78ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [110ms] refreshing cache from master, [111ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [113ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [115ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [118ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [118ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [451ms] refreshing cache from master, [451ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [453ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [455ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [456ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [457ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [2050ms] refreshing cache from master, [2051ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [2053ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [2056ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [2057ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [2058ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [4551ms] refreshing cache from master, [4551ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [4554ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@1175099903(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@6bc1e83 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1132417316 after retry RPC@1997100714) -> (continuation of Deferred@1735581545 after retry RPC@1516425819) -> (continuation of Deferred@586524000 after retry RPC@1196136272) -> (continuation of Deferred@1649529942 after retry RPC@812061348) -> (continuation of Deferred@1476746041 after retry RPC@1358921903) -> (continuation of Deferred@276311828 after retry RPC@1018048332), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@44934c17 -> wakeup thread Time-limited test -> (continuation of Deferred@1132417316 after retry RPC@1997100714) -> (continuation of Deferred@1735581545 after retry RPC@1516425819) -> (continuation of Deferred@586524000 after retry RPC@1196136272) -> (continuation of Deferred@1649529942 after retry RPC@812061348) -> (continuation of Deferred@1476746041 after retry RPC@1358921903) -> (continuation of Deferred@276311828 after retry RPC@1018048332))) after lookup
02:07:15.195 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] connecting to peer
02:07:15.197 [DEBUG - kudu-nio-2] (Connection.java:757) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] Successfully connected to peer
02:07:15.221 [DEBUG - kudu-nio-2] (Negotiator.java:919) Authenticated connection [id: 0x9d0f68f9, L:/127.0.0.1:41480 - R:/127.22.115.2:43855] using TOKEN/null
02:07:15.233 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.22.115.3:43855
02:07:15.234 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

02:07:15.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.235070 22988 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskTHXZc_/build/release/bin/kudu
02:07:15.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskTHXZc_/build/release/bin/kudu
02:07:15.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/wal
02:07:15.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/data
02:07:15.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:07:15.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:07:15.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:07:15.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:15.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:15.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:15.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/logs
02:07:15.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/data/info.pb
02:07:15.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:15.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:07:15.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:07:15.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:07:15.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:07:15.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:15.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:15.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:15.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:07:15.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:07:15.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.115.3:43855
02:07:15.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.115.3
02:07:15.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.115.62:38055,127.22.115.61:46503,127.22.115.60:37831
02:07:15.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45263
02:07:15.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.115.3
02:07:15.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.115.20:33307
02:07:15.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:15.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:15.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:07:15.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:15.313550 23774 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:07:15.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:15.313715 23774 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:07:15.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:15.313742 23774 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:07:15.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:15.315212 23774 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:07:15.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:15.315261 23774 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.115.3
02:07:15.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.316843 23774 tablet_server_runner.cc:78] Tablet server non-default flags:
02:07:15.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:15.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.115.20:33307
02:07:15.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:15.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:07:15.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/data
02:07:15.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/wal
02:07:15.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:15.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:15.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.115.3:43855
02:07:15.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:07:15.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:15.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:15.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/data/info.pb
02:07:15.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.115.3
02:07:15.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45263
02:07:15.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.115.62:38055,127.22.115.61:46503,127.22.115.60:37831
02:07:15.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:07:15.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.23774
02:07:15.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:15.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:07:15.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:07:15.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:15.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.115.3
02:07:15.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/logs
02:07:15.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:15.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:07:15.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:07:15.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:07:15.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:07:15.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
02:07:15.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
02:07:15.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 01:43:15 UTC on e7f111948823
02:07:15.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11949
02:07:15.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.317049 23774 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:07:15.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.317236 23774 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:07:15.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:15.319662 23779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:15.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:15.319653 23780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:15.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:15.319983 23782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:15.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.320652 23774 server_base.cc:1061] running on GCE node
02:07:15.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.320942 23774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:07:15.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.321195 23774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:07:15.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.322351 23774 hybrid_clock.cc:648] HybridClock initialized: now 1779070035322328 us; error 39 us; skew 500 ppm
02:07:15.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.323401 23774 webserver.cc:533] Webserver started at http://127.22.115.3:45263/ using document root <none> and password file <none>
02:07:15.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.323550 23774 fs_manager.cc:362] Metadata directory not provided
02:07:15.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.323601 23774 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:07:15.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.324790 23774 fs_manager.cc:714] Time spent opening directory manager: real 0.001s	user 0.001s	sys 0.000s
02:07:15.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.325462 23788 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:07:15.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.325889 23774 fs_manager.cc:730] Time spent opening block manager: real 0.001s	user 0.001s	sys 0.000s
02:07:15.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.326037 23774 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/data,/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/wal
02:07:15.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "174330aa0d3e49c69cab90b20e4f7a56"
02:07:15.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:07:09 on dist-test-slave-37fl"
02:07:15.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.326364 23774 fs_report.cc:389] FS layout report
02:07:15.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:07:15.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/wal
02:07:15.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/wal
02:07:15.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/data/data
02:07:15.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:07:15.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:07:15.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:07:15.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:07:15.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:07:15.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:07:15.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:07:15.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:07:15.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:07:15.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:07:15.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:07:15.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.364842 23774 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:07:15.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.365238 23774 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:07:15.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.365366 23774 kserver.cc:163] Server-wide thread pool size limit: 3276
02:07:15.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.365608 23774 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:07:15.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.366096 23795 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
02:07:15.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.366900 23774 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:07:15.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.366998 23774 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s	user 0.000s	sys 0.000s
02:07:15.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.367043 23774 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:07:15.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.367727 23774 ts_tablet_manager.cc:616] Registered 1 tablets
02:07:15.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.367794 23774 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s	user 0.000s	sys 0.000s
02:07:15.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.367815 23795 tablet_bootstrap.cc:492] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56: Bootstrap starting.
02:07:15.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.376338 23795 log.cc:826] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56: Log is configured to *not* fsync() on all Append() calls
02:07:15.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.376384 23774 rpc_server.cc:307] RPC server started. Bound to: 127.22.115.3:43855
02:07:15.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.376906 23774 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/data/info.pb
02:07:15.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.377297 23902 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.115.3:43855 every 8 connection(s)
02:07:15.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.380241 22988 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskTHXZc_/build/release/bin/kudu as pid 23774
02:07:15.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.380282 23795 tablet_bootstrap.cc:492] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:07:15.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.380590 23795 tablet_bootstrap.cc:492] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56: Bootstrap complete.
02:07:15.381 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:07:15.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.381803 23795 ts_tablet_manager.cc:1403] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56: Time spent bootstrapping tablet: real 0.014s	user 0.007s	sys 0.004s
02:07:15.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.383101 23795 raft_consensus.cc:359] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:15.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.383404 23795 raft_consensus.cc:740] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 174330aa0d3e49c69cab90b20e4f7a56, State: Initialized, Role: FOLLOWER
02:07:15.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.383504 23795 consensus_queue.cc:260] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:15.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.383754 23795 ts_tablet_manager.cc:1434] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
02:07:15.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.384557 23903 heartbeater.cc:344] Connected to a master server at 127.22.115.62:38055
02:07:15.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.384658 23903 heartbeater.cc:461] Registering TS with master...
02:07:15.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.384859 23903 heartbeater.cc:507] Master 127.22.115.62:38055 requested a full tablet report, sending...
02:07:15.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.385624 23028 ts_manager.cc:194] Re-registered known tserver with Master: 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855)
02:07:15.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.385962 23904 heartbeater.cc:344] Connected to a master server at 127.22.115.60:37831
02:07:15.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.386026 23904 heartbeater.cc:461] Registering TS with master...
02:07:15.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.386130 23904 heartbeater.cc:507] Master 127.22.115.60:37831 requested a full tablet report, sending...
02:07:15.386 [DEBUG - kudu-nio-2] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 6 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563, tablet=76ab850ff70d4ccca2590ffea11986ed, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:07:15.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.386451 23173 ts_manager.cc:194] Re-registered known tserver with Master: 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855)
02:07:15.387 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.22.115.2:43855
02:07:15.387 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

02:07:15.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.388041 22988 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskTHXZc_/build/release/bin/kudu with pid 23408
02:07:15.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.388037 23905 heartbeater.cc:344] Connected to a master server at 127.22.115.61:46503
02:07:15.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.388098 23905 heartbeater.cc:461] Registering TS with master...
02:07:15.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.388192 23905 heartbeater.cc:507] Master 127.22.115.61:46503 requested a full tablet report, sending...
02:07:15.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.390139 23101 ts_manager.cc:194] Re-registered known tserver with Master: 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855)
02:07:15.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.390551 23101 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.115.3:34005
02:07:15.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:15.392015 23905 heartbeater.cc:499] Master 127.22.115.61:46503 was elected leader, sending a full tablet report...
02:07:15.394 [DEBUG - kudu-nio-2] (Connection.java:239) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] handling channelInactive
02:07:15.395 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] cleaning up while in state READY due to: connection closed
02:07:15.395 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:07:15.396 [DEBUG - Time-limited test] (Connection.java:748) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] connecting to peer
02:07:15.398 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:15.398 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] cleaning up while in state CONNECTING due to: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:15.399 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 85635564f2c94c458f506c870a826b34(127.22.115.2:43855) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:15.413 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:15.413 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855),619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047),85635564f2c94c458f506c870a826b34(127.22.115.2:43855)[L]]}]
02:07:15.414 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=2, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=18), Traces: [0ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [3ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [3ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [16ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@1918402640(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@2330d6b5 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7375c83e -> wakeup thread Time-limited test)) after lookup
02:07:15.414 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] connecting to peer
02:07:15.416 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:15.416 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] cleaning up while in state CONNECTING due to: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:15.416 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 85635564f2c94c458f506c870a826b34(127.22.115.2:43855) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:15.433 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:15.433 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855),619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047),85635564f2c94c458f506c870a826b34(127.22.115.2:43855)[L]]}]
02:07:15.434 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=4, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=38), Traces: [0ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [3ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [3ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [16ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [18ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [20ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [20ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [37ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@1918402640(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@2330d6b5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2078829982 after retry RPC@637628133), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7375c83e -> wakeup thread Time-limited test -> (continuation of Deferred@2078829982 after retry RPC@637628133))) after lookup
02:07:15.435 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] connecting to peer
02:07:15.436 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:15.436 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] cleaning up while in state CONNECTING due to: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:15.436 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 85635564f2c94c458f506c870a826b34(127.22.115.2:43855) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:15.452 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:15.452 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855),619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047),85635564f2c94c458f506c870a826b34(127.22.115.2:43855)[L]]}]
02:07:15.453 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=6, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=56), Traces: [0ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [3ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [3ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [16ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [18ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [20ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [20ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [37ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [38ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [40ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [40ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [53ms] refreshing cache from master, [53ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [55ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@1918402640(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@2330d6b5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2078829982 after retry RPC@637628133) -> (continuation of Deferred@1245659602 after retry RPC@1743022134), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7375c83e -> wakeup thread Time-limited test -> (continuation of Deferred@2078829982 after retry RPC@637628133) -> (continuation of Deferred@1245659602 after retry RPC@1743022134))) after lookup
02:07:15.454 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] connecting to peer
02:07:15.455 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:15.455 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] cleaning up while in state CONNECTING due to: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:15.455 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 85635564f2c94c458f506c870a826b34(127.22.115.2:43855) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:15.552 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:15.553 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855),619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047),85635564f2c94c458f506c870a826b34(127.22.115.2:43855)[L]]}]
02:07:15.554 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=8, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=157), Traces: [0ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [3ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [3ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [16ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [18ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [20ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [20ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [37ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [38ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [40ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [40ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [53ms] refreshing cache from master, [53ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [55ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [58ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [59ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [59ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [154ms] refreshing cache from master, [154ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [156ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@1918402640(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@2330d6b5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2078829982 after retry RPC@637628133) -> (continuation of Deferred@1245659602 after retry RPC@1743022134) -> (continuation of Deferred@999355600 after retry RPC@2138152524), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7375c83e -> wakeup thread Time-limited test -> (continuation of Deferred@2078829982 after retry RPC@637628133) -> (continuation of Deferred@1245659602 after retry RPC@1743022134) -> (continuation of Deferred@999355600 after retry RPC@2138152524))) after lookup
02:07:15.555 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] connecting to peer
02:07:15.556 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:15.557 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] cleaning up while in state CONNECTING due to: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:15.557 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 85635564f2c94c458f506c870a826b34(127.22.115.2:43855) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:15.652 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:15.652 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855),619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047),85635564f2c94c458f506c870a826b34(127.22.115.2:43855)[L]]}]
02:07:15.653 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=10, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=256), Traces: [0ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [3ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [3ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [16ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [18ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [20ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [20ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [37ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [38ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [40ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [40ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [53ms] refreshing cache from master, [53ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [55ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [58ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [59ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [59ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [154ms] refreshing cache from master, [154ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [156ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [158ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [161ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [161ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [253ms] refreshing cache from master, [253ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [255ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@1918402640(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@2330d6b5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2078829982 after retry RPC@637628133) -> (continuation of Deferred@1245659602 after retry RPC@1743022134) -> (continuation of Deferred@999355600 after retry RPC@2138152524) -> (continuation of Deferred@281902300 after retry RPC@1497970810), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7375c83e -> wakeup thread Time-limited test -> (continuation of Deferred@2078829982 after retry RPC@637628133) -> (continuation of Deferred@1245659602 after retry RPC@1743022134) -> (continuation of Deferred@999355600 after retry RPC@2138152524) -> (continuation of Deferred@281902300 after retry RPC@1497970810))) after lookup
02:07:15.653 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] connecting to peer
02:07:15.655 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:15.655 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] cleaning up while in state CONNECTING due to: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:15.655 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 85635564f2c94c458f506c870a826b34(127.22.115.2:43855) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:16.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:16.732829 23917 raft_consensus.cc:493] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 85635564f2c94c458f506c870a826b34)
02:07:16.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:16.732932 23917 raft_consensus.cc:515] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:16.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:16.733109 23917 leader_election.cc:290] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 85635564f2c94c458f506c870a826b34 (127.22.115.2:43855), 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855)
02:07:16.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:16.733899 23291 leader_election.cc:336] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 85635564f2c94c458f506c870a826b34 (127.22.115.2:43855): Network error: Client connection negotiation failed: client connection to 127.22.115.2:43855: connect: Connection refused (error 111)
02:07:16.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:16.736447 23857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76ab850ff70d4ccca2590ffea11986ed" candidate_uuid: "619c08b8f3e246edabc0e4e315cf9201" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: false dest_uuid: "174330aa0d3e49c69cab90b20e4f7a56" is_pre_election: true
02:07:16.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:16.736583 23857 raft_consensus.cc:2468] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 619c08b8f3e246edabc0e4e315cf9201 in term 1.
02:07:16.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:16.736801 23288 leader_election.cc:304] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 174330aa0d3e49c69cab90b20e4f7a56, 619c08b8f3e246edabc0e4e315cf9201; no voters: 85635564f2c94c458f506c870a826b34
02:07:16.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:16.736909 23917 raft_consensus.cc:2804] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 3 FOLLOWER]: Leader pre-election won for term 4
02:07:16.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:16.736953 23917 raft_consensus.cc:493] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 85635564f2c94c458f506c870a826b34)
02:07:16.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:16.736994 23917 raft_consensus.cc:3060] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 3 FOLLOWER]: Advancing to term 4
02:07:16.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:16.737773 23917 raft_consensus.cc:515] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:16.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:16.737933 23917 leader_election.cc:290] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [CANDIDATE]: Term 4 election: Requested vote from peers 85635564f2c94c458f506c870a826b34 (127.22.115.2:43855), 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855)
02:07:16.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:16.738190 23857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76ab850ff70d4ccca2590ffea11986ed" candidate_uuid: "619c08b8f3e246edabc0e4e315cf9201" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: false dest_uuid: "174330aa0d3e49c69cab90b20e4f7a56"
02:07:16.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:16.738305 23857 raft_consensus.cc:3060] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 1 FOLLOWER]: Advancing to term 4
02:07:16.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:16.738428 23291 leader_election.cc:336] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 85635564f2c94c458f506c870a826b34 (127.22.115.2:43855): Network error: Client connection negotiation failed: client connection to 127.22.115.2:43855: connect: Connection refused (error 111)
02:07:16.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:16.739408 23857 raft_consensus.cc:2468] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 619c08b8f3e246edabc0e4e315cf9201 in term 4.
02:07:16.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:16.739671 23288 leader_election.cc:304] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 174330aa0d3e49c69cab90b20e4f7a56, 619c08b8f3e246edabc0e4e315cf9201; no voters: 85635564f2c94c458f506c870a826b34
02:07:16.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:16.739842 23917 raft_consensus.cc:2804] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 4 FOLLOWER]: Leader election won for term 4
02:07:16.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:16.740103 23917 raft_consensus.cc:697] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 4 LEADER]: Becoming Leader. State: Replica: 619c08b8f3e246edabc0e4e315cf9201, State: Running, Role: LEADER
02:07:16.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:16.740198 23917 consensus_queue.cc:237] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.8, Last appended by leader: 8, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:16.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:16.741076 23101 catalog_manager.cc:5696] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 reported cstate change: term changed from 3 to 4, leader changed from 85635564f2c94c458f506c870a826b34 (127.22.115.2) to 619c08b8f3e246edabc0e4e315cf9201 (127.22.115.1). New cstate: current_term: 4 leader_uuid: "619c08b8f3e246edabc0e4e315cf9201" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } health_report { overall_health: UNKNOWN } } }
02:07:17.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:17.167829 23291 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.115.2:43855: connect: Connection refused (error 111) [suppressed 3 similar messages]
02:07:17.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:17.168644 23291 consensus_peers.cc:597] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 -> Peer 85635564f2c94c458f506c870a826b34 (127.22.115.2:43855): Couldn't send request to peer 85635564f2c94c458f506c870a826b34. Status: Network error: Client connection negotiation failed: client connection to 127.22.115.2:43855: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:07:17.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.323611 23857 raft_consensus.cc:1275] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 4 FOLLOWER]: Refusing update from remote peer 619c08b8f3e246edabc0e4e315cf9201: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 9. (index mismatch)
02:07:17.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.323945 23917 consensus_queue.cc:1048] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [LEADER]: Connected to new peer: Peer: permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 4, Time since last communication: 0.000s
02:07:17.492 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:17.492 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855),619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)[L],85635564f2c94c458f506c870a826b34(127.22.115.2:43855)]}]
02:07:17.493 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=12, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=2096), Traces: [0ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [3ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [3ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [16ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [18ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [20ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [20ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [37ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [38ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [40ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [40ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [53ms] refreshing cache from master, [53ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [55ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [58ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [59ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [59ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [154ms] refreshing cache from master, [154ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [156ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [158ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [161ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [161ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [253ms] refreshing cache from master, [253ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [255ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [257ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [259ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [259ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [2093ms] refreshing cache from master, [2094ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [2096ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@1918402640(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@2330d6b5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2078829982 after retry RPC@637628133) -> (continuation of Deferred@1245659602 after retry RPC@1743022134) -> (continuation of Deferred@999355600 after retry RPC@2138152524) -> (continuation of Deferred@281902300 after retry RPC@1497970810) -> (continuation of Deferred@1377843385 after retry RPC@1119502447), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7375c83e -> wakeup thread Time-limited test -> (continuation of Deferred@2078829982 after retry RPC@637628133) -> (continuation of Deferred@1245659602 after retry RPC@1743022134) -> (continuation of Deferred@999355600 after retry RPC@2138152524) -> (continuation of Deferred@281902300 after retry RPC@1497970810) -> (continuation of Deferred@1377843385 after retry RPC@1119502447))) after lookup
02:07:17.493 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] connecting to peer
02:07:17.494 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] Successfully connected to peer
02:07:17.516 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0xcb3f3a68, L:/127.0.0.1:38792 - R:/127.22.115.1:46047] using TOKEN/null
02:07:17.527 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.22.115.2:43855
02:07:17.528 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

02:07:17.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.528926 22988 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskTHXZc_/build/release/bin/kudu
02:07:17.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskTHXZc_/build/release/bin/kudu
02:07:17.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/wal
02:07:17.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/data
02:07:17.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:07:17.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:07:17.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:07:17.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:17.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:17.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:17.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/logs
02:07:17.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/data/info.pb
02:07:17.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:17.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:07:17.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:07:17.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:07:17.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:07:17.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:17.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:17.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:17.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:07:17.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:07:17.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.115.2:43855
02:07:17.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.115.2
02:07:17.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.115.62:38055,127.22.115.61:46503,127.22.115.60:37831
02:07:17.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37975
02:07:17.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.115.2
02:07:17.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.115.20:33307
02:07:17.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:17.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:17.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:07:17.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:17.616181 23946 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:07:17.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:17.616346 23946 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:07:17.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:17.616366 23946 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:07:17.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:17.617785 23946 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:07:17.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:17.617832 23946 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.115.2
02:07:17.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.619434 23946 tablet_server_runner.cc:78] Tablet server non-default flags:
02:07:17.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:17.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.115.20:33307
02:07:17.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:17.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:07:17.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/data
02:07:17.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/wal
02:07:17.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:17.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:17.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.115.2:43855
02:07:17.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:07:17.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:17.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:17.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/data/info.pb
02:07:17.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.115.2
02:07:17.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37975
02:07:17.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.115.62:38055,127.22.115.61:46503,127.22.115.60:37831
02:07:17.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:07:17.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.23946
02:07:17.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:17.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:07:17.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:07:17.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:17.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.115.2
02:07:17.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/logs
02:07:17.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:17.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:07:17.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:07:17.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:07:17.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:07:17.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
02:07:17.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
02:07:17.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 01:43:15 UTC on e7f111948823
02:07:17.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11949
02:07:17.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.619668 23946 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:07:17.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.619879 23946 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:07:17.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:17.622354 23951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:17.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:17.622454 23952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:17.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.622692 23946 server_base.cc:1061] running on GCE node
02:07:17.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:17.622388 23954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:17.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.623005 23946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:07:17.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.623275 23946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:07:17.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.624418 23946 hybrid_clock.cc:648] HybridClock initialized: now 1779070037624389 us; error 44 us; skew 500 ppm
02:07:17.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.625571 23946 webserver.cc:533] Webserver started at http://127.22.115.2:37975/ using document root <none> and password file <none>
02:07:17.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.625794 23946 fs_manager.cc:362] Metadata directory not provided
02:07:17.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.625881 23946 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:07:17.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.627120 23946 fs_manager.cc:714] Time spent opening directory manager: real 0.001s	user 0.001s	sys 0.000s
02:07:17.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.627883 23960 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:07:17.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.628098 23946 fs_manager.cc:730] Time spent opening block manager: real 0.001s	user 0.000s	sys 0.000s
02:07:17.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.628194 23946 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/data,/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/wal
02:07:17.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "85635564f2c94c458f506c870a826b34"
02:07:17.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:07:09 on dist-test-slave-37fl"
02:07:17.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.628494 23946 fs_report.cc:389] FS layout report
02:07:17.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:07:17.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/wal
02:07:17.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/wal
02:07:17.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/data/data
02:07:17.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:07:17.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:07:17.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:07:17.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:07:17.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:07:17.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:07:17.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:07:17.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:07:17.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:07:17.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:07:17.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:07:17.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.653178 23946 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:07:17.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.653549 23946 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:07:17.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.653684 23946 kserver.cc:163] Server-wide thread pool size limit: 3276
02:07:17.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.653919 23946 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:07:17.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.654434 23967 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
02:07:17.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.655288 23946 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:07:17.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.655354 23946 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s	user 0.000s	sys 0.000s
02:07:17.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.655387 23946 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:07:17.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.655946 23946 ts_tablet_manager.cc:616] Registered 1 tablets
02:07:17.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.656062 23946 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s	user 0.001s	sys 0.000s
02:07:17.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.656100 23967 tablet_bootstrap.cc:492] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34: Bootstrap starting.
02:07:17.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.663472 23946 rpc_server.cc:307] RPC server started. Bound to: 127.22.115.2:43855
02:07:17.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.663571 24074 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.115.2:43855 every 8 connection(s)
02:07:17.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.663918 23946 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/data/info.pb
02:07:17.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.666224 23967 log.cc:826] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34: Log is configured to *not* fsync() on all Append() calls
02:07:17.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.669037 22988 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskTHXZc_/build/release/bin/kudu as pid 23946
02:07:17.669 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:07:17.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.669937 24076 heartbeater.cc:344] Connected to a master server at 127.22.115.60:37831
02:07:17.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.670318 24076 heartbeater.cc:461] Registering TS with master...
02:07:17.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.670521 24075 heartbeater.cc:344] Connected to a master server at 127.22.115.62:38055
02:07:17.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.670588 24075 heartbeater.cc:461] Registering TS with master...
02:07:17.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.670634 24077 heartbeater.cc:344] Connected to a master server at 127.22.115.61:46503
02:07:17.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.670684 24077 heartbeater.cc:461] Registering TS with master...
02:07:17.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.670728 24075 heartbeater.cc:507] Master 127.22.115.62:38055 requested a full tablet report, sending...
02:07:17.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.670778 24077 heartbeater.cc:507] Master 127.22.115.61:46503 requested a full tablet report, sending...
02:07:17.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.671459 24076 heartbeater.cc:507] Master 127.22.115.60:37831 requested a full tablet report, sending...
02:07:17.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.671890 23174 ts_manager.cc:194] Re-registered known tserver with Master: 85635564f2c94c458f506c870a826b34 (127.22.115.2:43855)
02:07:17.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.672312 23101 ts_manager.cc:194] Re-registered known tserver with Master: 85635564f2c94c458f506c870a826b34 (127.22.115.2:43855)
02:07:17.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.672648 23967 tablet_bootstrap.cc:492] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=6 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:07:17.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.672684 23101 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.115.2:53093
02:07:17.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.672976 23967 tablet_bootstrap.cc:492] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34: Bootstrap complete.
02:07:17.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.673856 23967 ts_tablet_manager.cc:1403] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34: Time spent bootstrapping tablet: real 0.018s	user 0.002s	sys 0.010s
02:07:17.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.674117 23029 ts_manager.cc:194] Re-registered known tserver with Master: 85635564f2c94c458f506c870a826b34 (127.22.115.2:43855)
02:07:17.675 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 9 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563, tablet=76ab850ff70d4ccca2590ffea11986ed, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:07:17.676 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.22.115.1:46047
02:07:17.676 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

02:07:17.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.676714 22988 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskTHXZc_/build/release/bin/kudu with pid 23273
02:07:17.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.680794 23967 raft_consensus.cc:359] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:17.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.680985 23967 raft_consensus.cc:740] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85635564f2c94c458f506c870a826b34, State: Initialized, Role: FOLLOWER
02:07:17.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.681099 23967 consensus_queue.cc:260] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 3.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:17.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.681373 23967 ts_tablet_manager.cc:1434] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.000s
02:07:17.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:17.681553 24077 heartbeater.cc:499] Master 127.22.115.61:46503 was elected leader, sending a full tablet report...
02:07:17.683 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:07:17.683 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] handling channelInactive
02:07:17.684 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] cleaning up while in state READY due to: connection closed
02:07:17.684 [DEBUG - Time-limited test] (Connection.java:748) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] connecting to peer
02:07:17.685 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:17.685 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] cleaning up while in state CONNECTING due to: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:17.685 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:17.691 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:17.692 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855),619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)[L],85635564f2c94c458f506c870a826b34(127.22.115.2:43855)]}]
02:07:17.692 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=2, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=8), Traces: [0ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [1ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [1ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [7ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@467413931(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@887415 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@78111d59 -> wakeup thread Time-limited test)) after lookup
02:07:17.693 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] connecting to peer
02:07:17.694 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:17.694 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] cleaning up while in state CONNECTING due to: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:17.694 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:17.711 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:17.712 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855),619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)[L],85635564f2c94c458f506c870a826b34(127.22.115.2:43855)]}]
02:07:17.712 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=4, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=28), Traces: [0ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [1ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [1ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [7ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [8ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [10ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [10ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [27ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@467413931(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@887415 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1694954998 after retry RPC@353224666), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@78111d59 -> wakeup thread Time-limited test -> (continuation of Deferred@1694954998 after retry RPC@353224666))) after lookup
02:07:17.713 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] connecting to peer
02:07:17.714 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:17.714 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] cleaning up while in state CONNECTING due to: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:17.714 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:17.753 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:17.753 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855),619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)[L],85635564f2c94c458f506c870a826b34(127.22.115.2:43855)]}]
02:07:17.753 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=6, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=69), Traces: [0ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [1ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [1ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [7ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [8ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [10ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [10ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [27ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [28ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [30ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [30ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [66ms] refreshing cache from master, [66ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [68ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@467413931(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@887415 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1694954998 after retry RPC@353224666) -> (continuation of Deferred@1650632938 after retry RPC@422219883), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@78111d59 -> wakeup thread Time-limited test -> (continuation of Deferred@1694954998 after retry RPC@353224666) -> (continuation of Deferred@1650632938 after retry RPC@422219883))) after lookup
02:07:17.754 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] connecting to peer
02:07:17.755 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:17.755 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] cleaning up while in state CONNECTING due to: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:17.756 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:17.872 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:17.872 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855),619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)[L],85635564f2c94c458f506c870a826b34(127.22.115.2:43855)]}]
02:07:17.872 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=8, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=188), Traces: [0ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [1ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [1ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [7ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [8ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [10ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [10ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [27ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [28ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [30ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [30ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [66ms] refreshing cache from master, [66ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [68ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [70ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [71ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [72ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [185ms] refreshing cache from master, [185ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [187ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@467413931(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@887415 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1694954998 after retry RPC@353224666) -> (continuation of Deferred@1650632938 after retry RPC@422219883) -> (continuation of Deferred@1184758999 after retry RPC@200906151), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@78111d59 -> wakeup thread Time-limited test -> (continuation of Deferred@1694954998 after retry RPC@353224666) -> (continuation of Deferred@1650632938 after retry RPC@422219883) -> (continuation of Deferred@1184758999 after retry RPC@200906151))) after lookup
02:07:17.873 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] connecting to peer
02:07:17.874 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:17.874 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] cleaning up while in state CONNECTING due to: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:17.874 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:18.012 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:18.012 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855),619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)[L],85635564f2c94c458f506c870a826b34(127.22.115.2:43855)]}]
02:07:18.013 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=10, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=329), Traces: [0ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [1ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [1ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [7ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [8ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [10ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [10ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [27ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [28ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [30ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [30ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [66ms] refreshing cache from master, [66ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [68ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [70ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [71ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [72ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [185ms] refreshing cache from master, [185ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [187ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [189ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [190ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [190ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [326ms] refreshing cache from master, [326ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [328ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@467413931(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@887415 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1694954998 after retry RPC@353224666) -> (continuation of Deferred@1650632938 after retry RPC@422219883) -> (continuation of Deferred@1184758999 after retry RPC@200906151) -> (continuation of Deferred@1412021915 after retry RPC@322587924), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@78111d59 -> wakeup thread Time-limited test -> (continuation of Deferred@1694954998 after retry RPC@353224666) -> (continuation of Deferred@1650632938 after retry RPC@422219883) -> (continuation of Deferred@1184758999 after retry RPC@200906151) -> (continuation of Deferred@1412021915 after retry RPC@322587924))) after lookup
02:07:18.014 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] connecting to peer
02:07:18.015 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:18.015 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] cleaning up while in state CONNECTING due to: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:18.015 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:18.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:18.377254 23898 messenger.cc:194] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.22.115.3:43855
02:07:18.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:18.664809 24070 messenger.cc:194] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.22.115.2:43855
02:07:18.832 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:18.832 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855),619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)[L],85635564f2c94c458f506c870a826b34(127.22.115.2:43855)]}]
02:07:18.833 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=12, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=1148), Traces: [0ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [1ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [1ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [7ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [8ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [10ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [10ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [27ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [28ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [30ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [30ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [66ms] refreshing cache from master, [66ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [68ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [70ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [71ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [72ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [185ms] refreshing cache from master, [185ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [187ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [189ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [190ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [190ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [326ms] refreshing cache from master, [326ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [328ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [329ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [331ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [332ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [1145ms] refreshing cache from master, [1145ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [1147ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@467413931(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@887415 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1694954998 after retry RPC@353224666) -> (continuation of Deferred@1650632938 after retry RPC@422219883) -> (continuation of Deferred@1184758999 after retry RPC@200906151) -> (continuation of Deferred@1412021915 after retry RPC@322587924) -> (continuation of Deferred@130251025 after retry RPC@1179698704), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@78111d59 -> wakeup thread Time-limited test -> (continuation of Deferred@1694954998 after retry RPC@353224666) -> (continuation of Deferred@1650632938 after retry RPC@422219883) -> (continuation of Deferred@1184758999 after retry RPC@200906151) -> (continuation of Deferred@1412021915 after retry RPC@322587924) -> (continuation of Deferred@130251025 after retry RPC@1179698704))) after lookup
02:07:18.833 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] connecting to peer
02:07:18.834 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:18.834 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] cleaning up while in state CONNECTING due to: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:18.835 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:19.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:19.027285 24089 raft_consensus.cc:493] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 619c08b8f3e246edabc0e4e315cf9201)
02:07:19.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:19.027379 24089 raft_consensus.cc:515] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:19.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:19.027704 24089 leader_election.cc:290] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 619c08b8f3e246edabc0e4e315cf9201 (127.22.115.1:46047), 85635564f2c94c458f506c870a826b34 (127.22.115.2:43855)
02:07:19.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:19.028237 23792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.115.1:46047: connect: Connection refused (error 111)
02:07:19.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:19.028723 23792 leader_election.cc:336] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 619c08b8f3e246edabc0e4e315cf9201 (127.22.115.1:46047): Network error: Client connection negotiation failed: client connection to 127.22.115.1:46047: connect: Connection refused (error 111)
02:07:19.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:19.031388 24029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76ab850ff70d4ccca2590ffea11986ed" candidate_uuid: "174330aa0d3e49c69cab90b20e4f7a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 12 } } ignore_live_leader: false dest_uuid: "85635564f2c94c458f506c870a826b34" is_pre_election: true
02:07:19.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:19.031518 24029 raft_consensus.cc:2468] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 174330aa0d3e49c69cab90b20e4f7a56 in term 3.
02:07:19.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:19.031742 23792 leader_election.cc:304] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 174330aa0d3e49c69cab90b20e4f7a56, 85635564f2c94c458f506c870a826b34; no voters: 619c08b8f3e246edabc0e4e315cf9201
02:07:19.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:19.031858 24089 raft_consensus.cc:2804] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 4 FOLLOWER]: Leader pre-election won for term 5
02:07:19.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:19.031916 24089 raft_consensus.cc:493] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 619c08b8f3e246edabc0e4e315cf9201)
02:07:19.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:19.032006 24089 raft_consensus.cc:3060] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 4 FOLLOWER]: Advancing to term 5
02:07:19.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:19.032598 24089 raft_consensus.cc:515] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:19.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:19.032732 24089 leader_election.cc:290] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [CANDIDATE]: Term 5 election: Requested vote from peers 619c08b8f3e246edabc0e4e315cf9201 (127.22.115.1:46047), 85635564f2c94c458f506c870a826b34 (127.22.115.2:43855)
02:07:19.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:19.033035 24029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76ab850ff70d4ccca2590ffea11986ed" candidate_uuid: "174330aa0d3e49c69cab90b20e4f7a56" candidate_term: 5 candidate_status { last_received { term: 4 index: 12 } } ignore_live_leader: false dest_uuid: "85635564f2c94c458f506c870a826b34"
02:07:19.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:19.033135 24029 raft_consensus.cc:3060] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 3 FOLLOWER]: Advancing to term 5
02:07:19.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:19.033327 23792 leader_election.cc:336] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 619c08b8f3e246edabc0e4e315cf9201 (127.22.115.1:46047): Network error: Client connection negotiation failed: client connection to 127.22.115.1:46047: connect: Connection refused (error 111)
02:07:19.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:19.034241 24029 raft_consensus.cc:2468] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 174330aa0d3e49c69cab90b20e4f7a56 in term 5.
02:07:19.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:19.034482 23792 leader_election.cc:304] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 174330aa0d3e49c69cab90b20e4f7a56, 85635564f2c94c458f506c870a826b34; no voters: 619c08b8f3e246edabc0e4e315cf9201
02:07:19.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:19.034593 24089 raft_consensus.cc:2804] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 5 FOLLOWER]: Leader election won for term 5
02:07:19.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:19.034713 24089 raft_consensus.cc:697] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 5 LEADER]: Becoming Leader. State: Replica: 174330aa0d3e49c69cab90b20e4f7a56, State: Running, Role: LEADER
02:07:19.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:19.034770 24089 consensus_queue.cc:237] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 4.12, Last appended by leader: 12, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:19.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:19.035509 23101 catalog_manager.cc:5696] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 reported cstate change: term changed from 4 to 5, leader changed from 619c08b8f3e246edabc0e4e315cf9201 (127.22.115.1) to 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3). New cstate: current_term: 5 leader_uuid: "174330aa0d3e49c69cab90b20e4f7a56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } health_report { overall_health: HEALTHY } } }
02:07:19.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:19.595504 24029 raft_consensus.cc:1275] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 5 FOLLOWER]: Refusing update from remote peer 174330aa0d3e49c69cab90b20e4f7a56: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 5 index: 13. (index mismatch)
02:07:19.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:19.595999 24106 consensus_queue.cc:1048] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 8, Time since last communication: 0.000s
02:07:19.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:19.599148 23792 consensus_peers.cc:597] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 -> Peer 619c08b8f3e246edabc0e4e315cf9201 (127.22.115.1:46047): Couldn't send request to peer 619c08b8f3e246edabc0e4e315cf9201. Status: Network error: Client connection negotiation failed: client connection to 127.22.115.1:46047: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:07:20.452 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:20.453 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)[L],619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047),85635564f2c94c458f506c870a826b34(127.22.115.2:43855)]}]
02:07:20.453 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=14, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=2769), Traces: [0ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [1ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [1ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [7ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [8ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [10ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [10ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [27ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [28ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [30ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [30ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [66ms] refreshing cache from master, [66ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [68ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [70ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [71ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [72ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [185ms] refreshing cache from master, [185ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [187ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [189ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [190ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [190ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [326ms] refreshing cache from master, [326ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [328ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [329ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [331ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [332ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [1145ms] refreshing cache from master, [1145ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [1147ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [1149ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [1150ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [1151ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [2766ms] refreshing cache from master, [2766ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [2768ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@467413931(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@887415 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1694954998 after retry RPC@353224666) -> (continuation of Deferred@1650632938 after retry RPC@422219883) -> (continuation of Deferred@1184758999 after retry RPC@200906151) -> (continuation of Deferred@1412021915 after retry RPC@322587924) -> (continuation of Deferred@130251025 after retry RPC@1179698704) -> (continuation of Deferred@38534084 after retry RPC@1114361307), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@78111d59 -> wakeup thread Time-limited test -> (continuation of Deferred@1694954998 after retry RPC@353224666) -> (continuation of Deferred@1650632938 after retry RPC@422219883) -> (continuation of Deferred@1184758999 after retry RPC@200906151) -> (continuation of Deferred@1412021915 after retry RPC@322587924) -> (continuation of Deferred@130251025 after retry RPC@1179698704) -> (continuation of Deferred@38534084 after retry RPC@1114361307))) after lookup
02:07:20.454 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] connecting to peer
02:07:20.455 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] Successfully connected to peer
02:07:20.475 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x5bea4cf7, L:/127.0.0.1:54348 - R:/127.22.115.3:43855] using TOKEN/null
02:07:20.486 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.22.115.1:46047
02:07:20.487 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

02:07:20.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.487463 22988 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskTHXZc_/build/release/bin/kudu
02:07:20.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskTHXZc_/build/release/bin/kudu
02:07:20.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/wal
02:07:20.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/data
02:07:20.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:07:20.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:07:20.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:07:20.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:20.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:20.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:20.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/logs
02:07:20.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/data/info.pb
02:07:20.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:20.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:07:20.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:07:20.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:07:20.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:07:20.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:20.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:20.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:20.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:07:20.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:07:20.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.115.1:46047
02:07:20.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.115.1
02:07:20.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.115.62:38055,127.22.115.61:46503,127.22.115.60:37831
02:07:20.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=35073
02:07:20.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.115.1
02:07:20.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.115.20:33307
02:07:20.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:20.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:20.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:07:20.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:20.566421 24126 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:07:20.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:20.566627 24126 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:07:20.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:20.566660 24126 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:07:20.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:20.568478 24126 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:07:20.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:20.568563 24126 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.115.1
02:07:20.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.570341 24126 tablet_server_runner.cc:78] Tablet server non-default flags:
02:07:20.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:20.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.115.20:33307
02:07:20.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:20.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:07:20.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/data
02:07:20.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/wal
02:07:20.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:20.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:20.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.115.1:46047
02:07:20.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:07:20.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:20.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:20.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/data/info.pb
02:07:20.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.115.1
02:07:20.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=35073
02:07:20.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.115.62:38055,127.22.115.61:46503,127.22.115.60:37831
02:07:20.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:07:20.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.24126
02:07:20.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:20.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:07:20.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:07:20.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:20.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.115.1
02:07:20.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/logs
02:07:20.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:20.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:07:20.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:07:20.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:07:20.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:07:20.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
02:07:20.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
02:07:20.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 01:43:15 UTC on e7f111948823
02:07:20.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11949
02:07:20.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.570614 24126 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:07:20.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.570880 24126 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:07:20.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:20.574301 24134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:20.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:20.574291 24132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:20.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:20.574376 24131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:20.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.574427 24126 server_base.cc:1061] running on GCE node
02:07:20.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.574854 24126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:07:20.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.575053 24126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:07:20.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.576251 24126 hybrid_clock.cc:648] HybridClock initialized: now 1779070040576226 us; error 41 us; skew 500 ppm
02:07:20.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.577428 24126 webserver.cc:533] Webserver started at http://127.22.115.1:35073/ using document root <none> and password file <none>
02:07:20.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.577584 24126 fs_manager.cc:362] Metadata directory not provided
02:07:20.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.577630 24126 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:07:20.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.578779 24126 fs_manager.cc:714] Time spent opening directory manager: real 0.001s	user 0.001s	sys 0.000s
02:07:20.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.579500 24140 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:07:20.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.579686 24126 fs_manager.cc:730] Time spent opening block manager: real 0.001s	user 0.001s	sys 0.000s
02:07:20.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.579741 24126 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/data,/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/wal
02:07:20.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "619c08b8f3e246edabc0e4e315cf9201"
02:07:20.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:07:08 on dist-test-slave-37fl"
02:07:20.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.580082 24126 fs_report.cc:389] FS layout report
02:07:20.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:07:20.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/wal
02:07:20.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/wal
02:07:20.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/data/data
02:07:20.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:07:20.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:07:20.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:07:20.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:07:20.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:07:20.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:07:20.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:07:20.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:07:20.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:07:20.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:07:20.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:07:20.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.592185 24126 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:07:20.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.592458 24126 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:07:20.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.592545 24126 kserver.cc:163] Server-wide thread pool size limit: 3276
02:07:20.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.592711 24126 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:07:20.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.593123 24147 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
02:07:20.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.594065 24126 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:07:20.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.594122 24126 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s	user 0.000s	sys 0.000s
02:07:20.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.594151 24126 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:07:20.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.594709 24126 ts_tablet_manager.cc:616] Registered 1 tablets
02:07:20.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.594741 24126 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s	user 0.000s	sys 0.000s
02:07:20.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.594861 24147 tablet_bootstrap.cc:492] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201: Bootstrap starting.
02:07:20.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.602550 24126 rpc_server.cc:307] RPC server started. Bound to: 127.22.115.1:46047
02:07:20.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.602597 24254 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.115.1:46047 every 8 connection(s)
02:07:20.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.603119 24126 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/data/info.pb
02:07:20.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.606493 24147 log.cc:826] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201: Log is configured to *not* fsync() on all Append() calls
02:07:20.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.609611 24147 tablet_bootstrap.cc:492] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=9 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:07:20.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.609872 24147 tablet_bootstrap.cc:492] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201: Bootstrap complete.
02:07:20.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.610667 24147 ts_tablet_manager.cc:1403] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201: Time spent bootstrapping tablet: real 0.016s	user 0.007s	sys 0.006s
02:07:20.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.611474 24147 raft_consensus.cc:359] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:20.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.611698 24147 raft_consensus.cc:740] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 619c08b8f3e246edabc0e4e315cf9201, State: Initialized, Role: FOLLOWER
02:07:20.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.611801 24147 consensus_queue.cc:260] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 4.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:20.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.612082 24147 ts_tablet_manager.cc:1434] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.002s
02:07:20.612 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:07:20.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.612279 22988 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskTHXZc_/build/release/bin/kudu as pid 24126
02:07:20.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.613337 24256 heartbeater.cc:344] Connected to a master server at 127.22.115.60:37831
02:07:20.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.613430 24256 heartbeater.cc:461] Registering TS with master...
02:07:20.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.613590 24256 heartbeater.cc:507] Master 127.22.115.60:37831 requested a full tablet report, sending...
02:07:20.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.614151 23174 ts_manager.cc:194] Re-registered known tserver with Master: 619c08b8f3e246edabc0e4e315cf9201 (127.22.115.1:46047)
02:07:20.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.616245 24257 heartbeater.cc:344] Connected to a master server at 127.22.115.61:46503
02:07:20.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.616318 24257 heartbeater.cc:461] Registering TS with master...
02:07:20.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.616453 24257 heartbeater.cc:507] Master 127.22.115.61:46503 requested a full tablet report, sending...
02:07:20.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.617913 23101 ts_manager.cc:194] Re-registered known tserver with Master: 619c08b8f3e246edabc0e4e315cf9201 (127.22.115.1:46047)
02:07:20.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.618461 23101 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.115.1:41925
02:07:20.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.619864 24255 heartbeater.cc:344] Connected to a master server at 127.22.115.62:38055
02:07:20.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.619937 24255 heartbeater.cc:461] Registering TS with master...
02:07:20.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.620131 24255 heartbeater.cc:507] Master 127.22.115.62:38055 requested a full tablet report, sending...
02:07:20.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.620627 23029 ts_manager.cc:194] Re-registered known tserver with Master: 619c08b8f3e246edabc0e4e315cf9201 (127.22.115.1:46047)
02:07:20.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.620775 24257 heartbeater.cc:499] Master 127.22.115.61:46503 was elected leader, sending a full tablet report...
02:07:20.621 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 12 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563, tablet=76ab850ff70d4ccca2590ffea11986ed, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:07:20.627 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.22.115.3:43855
02:07:20.628 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

02:07:20.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:20.628317 22988 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskTHXZc_/build/release/bin/kudu with pid 23774
02:07:20.634 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:07:20.634 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] handling channelInactive
02:07:20.634 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] cleaning up while in state READY due to: connection closed
02:07:20.650 [DEBUG - pool-2-thread-1] (Connection.java:748) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] connecting to peer
02:07:20.652 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:20.652 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] cleaning up while in state CONNECTING due to: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:20.652 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:20.672 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:20.672 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)[L],619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047),85635564f2c94c458f506c870a826b34(127.22.115.2:43855)]}]
02:07:20.673 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=3, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=38), Traces: [0ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [0ms] delaying RPC due to: Illegal state: connection is terminated, [16ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [18ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [18ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [38ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@108250025(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@5807f72 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@19431846 -> wakeup thread Time-limited test)) after lookup
02:07:20.673 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] connecting to peer
02:07:20.675 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:20.675 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] cleaning up while in state CONNECTING due to: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:20.675 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:20.691 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:20.692 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)[L],619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047),85635564f2c94c458f506c870a826b34(127.22.115.2:43855)]}]
02:07:20.692 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=5, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=58), Traces: [0ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [0ms] delaying RPC due to: Illegal state: connection is terminated, [16ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [18ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [18ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [38ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [39ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [41ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [41ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [55ms] refreshing cache from master, [55ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [57ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@108250025(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@5807f72 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1326098983 after retry RPC@1309285032), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@19431846 -> wakeup thread Time-limited test -> (continuation of Deferred@1326098983 after retry RPC@1309285032))) after lookup
02:07:20.693 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] connecting to peer
02:07:20.694 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:20.694 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] cleaning up while in state CONNECTING due to: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:20.694 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:20.732 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:20.732 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)[L],619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047),85635564f2c94c458f506c870a826b34(127.22.115.2:43855)]}]
02:07:20.733 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=7, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=98), Traces: [0ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [0ms] delaying RPC due to: Illegal state: connection is terminated, [16ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [18ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [18ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [38ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [39ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [41ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [41ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [55ms] refreshing cache from master, [55ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [57ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [59ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [60ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [60ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [96ms] refreshing cache from master, [96ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [98ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@108250025(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@5807f72 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1326098983 after retry RPC@1309285032) -> (continuation of Deferred@2070554038 after retry RPC@1705754409), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@19431846 -> wakeup thread Time-limited test -> (continuation of Deferred@1326098983 after retry RPC@1309285032) -> (continuation of Deferred@2070554038 after retry RPC@1705754409))) after lookup
02:07:20.734 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] connecting to peer
02:07:20.735 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:20.735 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] cleaning up while in state CONNECTING due to: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:20.735 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:20.932 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:20.933 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)[L],619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047),85635564f2c94c458f506c870a826b34(127.22.115.2:43855)]}]
02:07:20.933 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=9, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=299), Traces: [0ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [0ms] delaying RPC due to: Illegal state: connection is terminated, [16ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [18ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [18ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [38ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [39ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [41ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [41ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [55ms] refreshing cache from master, [55ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [57ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [59ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [60ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [60ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [96ms] refreshing cache from master, [96ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [98ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [99ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [101ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [101ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [296ms] refreshing cache from master, [296ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [298ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@108250025(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@5807f72 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1326098983 after retry RPC@1309285032) -> (continuation of Deferred@2070554038 after retry RPC@1705754409) -> (continuation of Deferred@1638070922 after retry RPC@1497908064), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@19431846 -> wakeup thread Time-limited test -> (continuation of Deferred@1326098983 after retry RPC@1309285032) -> (continuation of Deferred@2070554038 after retry RPC@1705754409) -> (continuation of Deferred@1638070922 after retry RPC@1497908064))) after lookup
02:07:20.934 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] connecting to peer
02:07:20.935 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:20.935 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] cleaning up while in state CONNECTING due to: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:20.935 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:21.592 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:21.592 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)[L],619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047),85635564f2c94c458f506c870a826b34(127.22.115.2:43855)]}]
02:07:21.593 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=11, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=958), Traces: [0ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [0ms] delaying RPC due to: Illegal state: connection is terminated, [16ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [18ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [18ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [38ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [39ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [41ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [41ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [55ms] refreshing cache from master, [55ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [57ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [59ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [60ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [60ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [96ms] refreshing cache from master, [96ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [98ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [99ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [101ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [101ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [296ms] refreshing cache from master, [296ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [298ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [300ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [301ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [301ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [955ms] refreshing cache from master, [955ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [958ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@108250025(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@5807f72 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1326098983 after retry RPC@1309285032) -> (continuation of Deferred@2070554038 after retry RPC@1705754409) -> (continuation of Deferred@1638070922 after retry RPC@1497908064) -> (continuation of Deferred@912506086 after retry RPC@757836297), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@19431846 -> wakeup thread Time-limited test -> (continuation of Deferred@1326098983 after retry RPC@1309285032) -> (continuation of Deferred@2070554038 after retry RPC@1705754409) -> (continuation of Deferred@1638070922 after retry RPC@1497908064) -> (continuation of Deferred@912506086 after retry RPC@757836297))) after lookup
02:07:21.594 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] connecting to peer
02:07:21.595 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:21.595 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] cleaning up while in state CONNECTING due to: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:21.595 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:21.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:21.941992 24268 raft_consensus.cc:493] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
02:07:21.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:21.942117 24268 raft_consensus.cc:515] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:21.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:21.942432 24268 leader_election.cc:290] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 85635564f2c94c458f506c870a826b34 (127.22.115.2:43855), 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855)
02:07:21.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:21.942934 24141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.115.3:43855: connect: Connection refused (error 111)
02:07:21.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:21.943593 24141 leader_election.cc:336] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855): Network error: Client connection negotiation failed: client connection to 127.22.115.3:43855: connect: Connection refused (error 111)
02:07:21.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:21.946086 24029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76ab850ff70d4ccca2590ffea11986ed" candidate_uuid: "619c08b8f3e246edabc0e4e315cf9201" candidate_term: 5 candidate_status { last_received { term: 4 index: 12 } } ignore_live_leader: false dest_uuid: "85635564f2c94c458f506c870a826b34" is_pre_election: true
02:07:21.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:21.946408 24144 leader_election.cc:304] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 619c08b8f3e246edabc0e4e315cf9201; no voters: 174330aa0d3e49c69cab90b20e4f7a56, 85635564f2c94c458f506c870a826b34
02:07:21.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:21.946556 24268 raft_consensus.cc:2749] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
02:07:21.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:21.986640 24273 raft_consensus.cc:493] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 174330aa0d3e49c69cab90b20e4f7a56)
02:07:21.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:21.986755 24273 raft_consensus.cc:515] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:21.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:21.987093 24273 leader_election.cc:290] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 619c08b8f3e246edabc0e4e315cf9201 (127.22.115.1:46047), 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855)
02:07:21.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:21.987555 23961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.115.3:43855: connect: Connection refused (error 111)
02:07:21.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:21.988216 23961 leader_election.cc:336] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855): Network error: Client connection negotiation failed: client connection to 127.22.115.3:43855: connect: Connection refused (error 111)
02:07:21.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:21.990469 24209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76ab850ff70d4ccca2590ffea11986ed" candidate_uuid: "85635564f2c94c458f506c870a826b34" candidate_term: 6 candidate_status { last_received { term: 5 index: 16 } } ignore_live_leader: false dest_uuid: "619c08b8f3e246edabc0e4e315cf9201" is_pre_election: true
02:07:21.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:21.990581 24209 raft_consensus.cc:2468] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85635564f2c94c458f506c870a826b34 in term 4.
02:07:21.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:21.990819 23964 leader_election.cc:304] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 619c08b8f3e246edabc0e4e315cf9201, 85635564f2c94c458f506c870a826b34; no voters: 174330aa0d3e49c69cab90b20e4f7a56
02:07:21.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:21.991011 24273 raft_consensus.cc:2804] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 5 FOLLOWER]: Leader pre-election won for term 6
02:07:21.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:21.991058 24273 raft_consensus.cc:493] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 174330aa0d3e49c69cab90b20e4f7a56)
02:07:21.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:21.991073 24273 raft_consensus.cc:3060] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 5 FOLLOWER]: Advancing to term 6
02:07:21.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:21.991720 24273 raft_consensus.cc:515] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:21.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:21.991858 24273 leader_election.cc:290] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [CANDIDATE]: Term 6 election: Requested vote from peers 619c08b8f3e246edabc0e4e315cf9201 (127.22.115.1:46047), 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855)
02:07:21.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:21.992038 24209 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76ab850ff70d4ccca2590ffea11986ed" candidate_uuid: "85635564f2c94c458f506c870a826b34" candidate_term: 6 candidate_status { last_received { term: 5 index: 16 } } ignore_live_leader: false dest_uuid: "619c08b8f3e246edabc0e4e315cf9201"
02:07:21.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:21.992122 24209 raft_consensus.cc:3060] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 4 FOLLOWER]: Advancing to term 6
02:07:21.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:21.992452 23961 leader_election.cc:336] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855): Network error: Client connection negotiation failed: client connection to 127.22.115.3:43855: connect: Connection refused (error 111)
02:07:21.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:21.993139 24209 raft_consensus.cc:2468] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85635564f2c94c458f506c870a826b34 in term 6.
02:07:21.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:21.993316 23964 leader_election.cc:304] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 619c08b8f3e246edabc0e4e315cf9201, 85635564f2c94c458f506c870a826b34; no voters: 174330aa0d3e49c69cab90b20e4f7a56
02:07:21.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:21.993409 24273 raft_consensus.cc:2804] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 6 FOLLOWER]: Leader election won for term 6
02:07:21.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:21.993546 24273 raft_consensus.cc:697] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 6 LEADER]: Becoming Leader. State: Replica: 85635564f2c94c458f506c870a826b34, State: Running, Role: LEADER
02:07:21.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:21.993611 24273 consensus_queue.cc:237] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 5.16, Last appended by leader: 16, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:21.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:21.994306 23101 catalog_manager.cc:5696] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 reported cstate change: term changed from 5 to 6, leader changed from 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3) to 85635564f2c94c458f506c870a826b34 (127.22.115.2). New cstate: current_term: 6 leader_uuid: "85635564f2c94c458f506c870a826b34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } health_report { overall_health: UNKNOWN } } }
02:07:22.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:22.416385 23961 consensus_peers.cc:597] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 -> Peer 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855): Couldn't send request to peer 174330aa0d3e49c69cab90b20e4f7a56. Status: Network error: Client connection negotiation failed: client connection to 127.22.115.3:43855: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:07:22.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:22.543578 24209 raft_consensus.cc:1275] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 6 FOLLOWER]: Refusing update from remote peer 85635564f2c94c458f506c870a826b34: Log matching property violated. Preceding OpId in replica: term: 4 index: 12. Preceding OpId from leader: term: 6 index: 17. (index mismatch)
02:07:22.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:22.544001 24273 consensus_queue.cc:1048] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 12, Time since last communication: 0.000s
02:07:23.752 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:23.753 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855),619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047),85635564f2c94c458f506c870a826b34(127.22.115.2:43855)[L]]}]
02:07:23.753 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=13, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=3118), Traces: [0ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [0ms] delaying RPC due to: Illegal state: connection is terminated, [16ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [18ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [18ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [38ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [39ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [41ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [41ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [55ms] refreshing cache from master, [55ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [57ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [59ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [60ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [60ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [96ms] refreshing cache from master, [96ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [98ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [99ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [101ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [101ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [296ms] refreshing cache from master, [296ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [298ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [300ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [301ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [301ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [955ms] refreshing cache from master, [955ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [958ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [959ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [961ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [961ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [3116ms] refreshing cache from master, [3116ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [3118ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@108250025(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@5807f72 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1326098983 after retry RPC@1309285032) -> (continuation of Deferred@2070554038 after retry RPC@1705754409) -> (continuation of Deferred@1638070922 after retry RPC@1497908064) -> (continuation of Deferred@912506086 after retry RPC@757836297) -> (continuation of Deferred@799792364 after retry RPC@1604342002), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@19431846 -> wakeup thread Time-limited test -> (continuation of Deferred@1326098983 after retry RPC@1309285032) -> (continuation of Deferred@2070554038 after retry RPC@1705754409) -> (continuation of Deferred@1638070922 after retry RPC@1497908064) -> (continuation of Deferred@912506086 after retry RPC@757836297) -> (continuation of Deferred@799792364 after retry RPC@1604342002))) after lookup
02:07:23.754 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] connecting to peer
02:07:23.755 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] Successfully connected to peer
02:07:23.773 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x82b72992, L:/127.0.0.1:49922 - R:/127.22.115.2:43855] using TOKEN/null
02:07:23.784 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.22.115.3:43855
02:07:23.784 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

02:07:23.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.785172 22988 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskTHXZc_/build/release/bin/kudu
02:07:23.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskTHXZc_/build/release/bin/kudu
02:07:23.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/wal
02:07:23.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/data
02:07:23.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:07:23.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:07:23.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:07:23.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:23.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:23.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:23.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/logs
02:07:23.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/data/info.pb
02:07:23.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:23.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:07:23.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:07:23.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:07:23.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:07:23.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:23.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:23.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:23.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:07:23.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:07:23.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.115.3:43855
02:07:23.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.115.3
02:07:23.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.115.62:38055,127.22.115.61:46503,127.22.115.60:37831
02:07:23.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45263
02:07:23.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.115.3
02:07:23.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.115.20:33307
02:07:23.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:23.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:23.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:07:23.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:23.865072 24310 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:07:23.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:23.865242 24310 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:07:23.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:23.865262 24310 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:07:23.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:23.866824 24310 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:07:23.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:23.866882 24310 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.115.3
02:07:23.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.868405 24310 tablet_server_runner.cc:78] Tablet server non-default flags:
02:07:23.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:23.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.115.20:33307
02:07:23.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:23.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:07:23.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/data
02:07:23.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/wal
02:07:23.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:23.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:23.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.115.3:43855
02:07:23.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:07:23.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:23.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:23.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/data/info.pb
02:07:23.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.115.3
02:07:23.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45263
02:07:23.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.115.62:38055,127.22.115.61:46503,127.22.115.60:37831
02:07:23.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:07:23.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.24310
02:07:23.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:23.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:07:23.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:07:23.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:23.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.115.3
02:07:23.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/logs
02:07:23.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:23.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:07:23.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:07:23.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:07:23.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:07:23.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
02:07:23.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
02:07:23.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 01:43:15 UTC on e7f111948823
02:07:23.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11949
02:07:23.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.868599 24310 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:07:23.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.868781 24310 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:07:23.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:23.871367 24318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:23.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:23.871431 24316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:23.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:23.871519 24315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:23.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.871601 24310 server_base.cc:1061] running on GCE node
02:07:23.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.871831 24310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:07:23.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.872092 24310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:07:23.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.873234 24310 hybrid_clock.cc:648] HybridClock initialized: now 1779070043873204 us; error 38 us; skew 500 ppm
02:07:23.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.874303 24310 webserver.cc:533] Webserver started at http://127.22.115.3:45263/ using document root <none> and password file <none>
02:07:23.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.874523 24310 fs_manager.cc:362] Metadata directory not provided
02:07:23.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.874594 24310 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:07:23.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.875744 24310 fs_manager.cc:714] Time spent opening directory manager: real 0.001s	user 0.000s	sys 0.001s
02:07:23.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.876403 24324 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:07:23.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.876575 24310 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.000s
02:07:23.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.876669 24310 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/data,/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/wal
02:07:23.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "174330aa0d3e49c69cab90b20e4f7a56"
02:07:23.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:07:09 on dist-test-slave-37fl"
02:07:23.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.876967 24310 fs_report.cc:389] FS layout report
02:07:23.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:07:23.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/wal
02:07:23.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/wal
02:07:23.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/data/data
02:07:23.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:07:23.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:07:23.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:07:23.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:07:23.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:07:23.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:07:23.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:07:23.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:07:23.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:07:23.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:07:23.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:07:23.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.886565 24310 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:07:23.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.886937 24310 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:07:23.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.887089 24310 kserver.cc:163] Server-wide thread pool size limit: 3276
02:07:23.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.887320 24310 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:07:23.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.887778 24331 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
02:07:23.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.888690 24310 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:07:23.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.888746 24310 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s	user 0.000s	sys 0.000s
02:07:23.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.888782 24310 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:07:23.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.889355 24310 ts_tablet_manager.cc:616] Registered 1 tablets
02:07:23.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.889410 24310 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s	user 0.000s	sys 0.000s
02:07:23.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.889454 24331 tablet_bootstrap.cc:492] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56: Bootstrap starting.
02:07:23.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.896644 24310 rpc_server.cc:307] RPC server started. Bound to: 127.22.115.3:43855
02:07:23.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.896718 24438 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.115.3:43855 every 8 connection(s)
02:07:23.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.897068 24310 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/data/info.pb
02:07:23.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.899856 22988 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskTHXZc_/build/release/bin/kudu as pid 24310
02:07:23.900 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:07:23.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.901464 24331 log.cc:826] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56: Log is configured to *not* fsync() on all Append() calls
02:07:23.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.903493 24440 heartbeater.cc:344] Connected to a master server at 127.22.115.60:37831
02:07:23.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.903596 24440 heartbeater.cc:461] Registering TS with master...
02:07:23.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.903802 24440 heartbeater.cc:507] Master 127.22.115.60:37831 requested a full tablet report, sending...
02:07:23.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.904551 24439 heartbeater.cc:344] Connected to a master server at 127.22.115.62:38055
02:07:23.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.904600 24439 heartbeater.cc:461] Registering TS with master...
02:07:23.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.904685 24439 heartbeater.cc:507] Master 127.22.115.62:38055 requested a full tablet report, sending...
02:07:23.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.904996 23028 ts_manager.cc:194] Re-registered known tserver with Master: 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855)
02:07:23.905 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 15 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563, tablet=76ab850ff70d4ccca2590ffea11986ed, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:07:23.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.905356 23174 ts_manager.cc:194] Re-registered known tserver with Master: 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855)
02:07:23.906 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.22.115.2:43855
02:07:23.906 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

02:07:23.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.906790 22988 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskTHXZc_/build/release/bin/kudu with pid 23946
02:07:23.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.908795 24331 tablet_bootstrap.cc:492] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=12 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:07:23.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.909235 24331 tablet_bootstrap.cc:492] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56: Bootstrap complete.
02:07:23.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.910327 24331 ts_tablet_manager.cc:1403] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56: Time spent bootstrapping tablet: real 0.021s	user 0.001s	sys 0.013s
02:07:23.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.910497 24441 heartbeater.cc:344] Connected to a master server at 127.22.115.61:46503
02:07:23.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.910550 24441 heartbeater.cc:461] Registering TS with master...
02:07:23.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.910665 24441 heartbeater.cc:507] Master 127.22.115.61:46503 requested a full tablet report, sending...
02:07:23.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.910938 23102 ts_manager.cc:194] Re-registered known tserver with Master: 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855)
02:07:23.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.911456 23102 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.115.3:34971
02:07:23.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.911580 24331 raft_consensus.cc:359] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:23.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.911842 24331 raft_consensus.cc:740] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 174330aa0d3e49c69cab90b20e4f7a56, State: Initialized, Role: FOLLOWER
02:07:23.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.911991 24331 consensus_queue.cc:260] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 5.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:23.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.912256 24331 ts_tablet_manager.cc:1434] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
02:07:23.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:23.912376 24441 heartbeater.cc:499] Master 127.22.115.61:46503 was elected leader, sending a full tablet report...
02:07:23.914 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] handling channelInactive
02:07:23.914 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] cleaning up while in state READY due to: connection closed
02:07:23.915 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:07:23.915 [DEBUG - Time-limited test] (Connection.java:748) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] connecting to peer
02:07:23.916 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:23.917 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] cleaning up while in state CONNECTING due to: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:23.917 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 85635564f2c94c458f506c870a826b34(127.22.115.2:43855) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:23.932 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:23.933 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855),619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047),85635564f2c94c458f506c870a826b34(127.22.115.2:43855)[L]]}]
02:07:23.933 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=2, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=18), Traces: [0ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [2ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [2ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [17ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@33065829(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@5c026c68 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@13724f78 -> wakeup thread Time-limited test)) after lookup
02:07:23.934 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] connecting to peer
02:07:23.935 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:23.935 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] cleaning up while in state CONNECTING due to: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:23.935 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 85635564f2c94c458f506c870a826b34(127.22.115.2:43855) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:23.952 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:23.952 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855),619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047),85635564f2c94c458f506c870a826b34(127.22.115.2:43855)[L]]}]
02:07:23.953 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=4, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=37), Traces: [0ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [2ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [2ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [17ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [19ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [20ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [20ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [34ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [37ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@33065829(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@5c026c68 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@641778548 after retry RPC@920173233), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@13724f78 -> wakeup thread Time-limited test -> (continuation of Deferred@641778548 after retry RPC@920173233))) after lookup
02:07:23.953 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] connecting to peer
02:07:23.954 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:23.954 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] cleaning up while in state CONNECTING due to: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:23.954 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 85635564f2c94c458f506c870a826b34(127.22.115.2:43855) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:23.972 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:23.972 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855),619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047),85635564f2c94c458f506c870a826b34(127.22.115.2:43855)[L]]}]
02:07:23.972 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=6, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=57), Traces: [0ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [2ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [2ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [17ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [19ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [20ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [20ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [34ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [37ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [38ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [39ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [39ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [54ms] refreshing cache from master, [54ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [56ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@33065829(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@5c026c68 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@641778548 after retry RPC@920173233) -> (continuation of Deferred@1402723071 after retry RPC@551943951), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@13724f78 -> wakeup thread Time-limited test -> (continuation of Deferred@641778548 after retry RPC@920173233) -> (continuation of Deferred@1402723071 after retry RPC@551943951))) after lookup
02:07:23.973 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] connecting to peer
02:07:23.974 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:23.974 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] cleaning up while in state CONNECTING due to: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:23.974 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 85635564f2c94c458f506c870a826b34(127.22.115.2:43855) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:24.112 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:24.112 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855),619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047),85635564f2c94c458f506c870a826b34(127.22.115.2:43855)[L]]}]
02:07:24.113 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=8, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=197), Traces: [0ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [2ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [2ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [17ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [19ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [20ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [20ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [34ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [37ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [38ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [39ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [39ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [54ms] refreshing cache from master, [54ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [56ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [58ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [59ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [59ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [194ms] refreshing cache from master, [195ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [197ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@33065829(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@5c026c68 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@641778548 after retry RPC@920173233) -> (continuation of Deferred@1402723071 after retry RPC@551943951) -> (continuation of Deferred@1627081167 after retry RPC@1869209432), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@13724f78 -> wakeup thread Time-limited test -> (continuation of Deferred@641778548 after retry RPC@920173233) -> (continuation of Deferred@1402723071 after retry RPC@551943951) -> (continuation of Deferred@1627081167 after retry RPC@1869209432))) after lookup
02:07:24.113 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] connecting to peer
02:07:24.114 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:24.114 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] cleaning up while in state CONNECTING due to: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:24.114 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 85635564f2c94c458f506c870a826b34(127.22.115.2:43855) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:24.412 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:24.412 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855),619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047),85635564f2c94c458f506c870a826b34(127.22.115.2:43855)[L]]}]
02:07:24.412 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=10, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=497), Traces: [0ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [2ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [2ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [17ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [19ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [20ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [20ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [34ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [37ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [38ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [39ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [39ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [54ms] refreshing cache from master, [54ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [56ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [58ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [59ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [59ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [194ms] refreshing cache from master, [195ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [197ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [198ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [199ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [200ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [494ms] refreshing cache from master, [494ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [496ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@33065829(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@5c026c68 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@641778548 after retry RPC@920173233) -> (continuation of Deferred@1402723071 after retry RPC@551943951) -> (continuation of Deferred@1627081167 after retry RPC@1869209432) -> (continuation of Deferred@599768419 after retry RPC@42808659), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@13724f78 -> wakeup thread Time-limited test -> (continuation of Deferred@641778548 after retry RPC@920173233) -> (continuation of Deferred@1402723071 after retry RPC@551943951) -> (continuation of Deferred@1627081167 after retry RPC@1869209432) -> (continuation of Deferred@599768419 after retry RPC@42808659))) after lookup
02:07:24.413 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] connecting to peer
02:07:24.414 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:24.415 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] cleaning up while in state CONNECTING due to: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:24.415 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 85635564f2c94c458f506c870a826b34(127.22.115.2:43855) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:24.451 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:24.452 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855),619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047),85635564f2c94c458f506c870a826b34(127.22.115.2:43855)[L]]}]
02:07:24.452 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=12, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=537), Traces: [0ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [2ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [2ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [17ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [19ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [20ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [20ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [34ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [37ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [38ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [39ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [39ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [54ms] refreshing cache from master, [54ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [56ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [58ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [59ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [59ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [194ms] refreshing cache from master, [195ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [197ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [198ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [199ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [200ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [494ms] refreshing cache from master, [494ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [496ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [498ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [500ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [500ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [534ms] refreshing cache from master, [534ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [536ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@33065829(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@5c026c68 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@641778548 after retry RPC@920173233) -> (continuation of Deferred@1402723071 after retry RPC@551943951) -> (continuation of Deferred@1627081167 after retry RPC@1869209432) -> (continuation of Deferred@599768419 after retry RPC@42808659) -> (continuation of Deferred@2134900209 after retry RPC@351500520), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@13724f78 -> wakeup thread Time-limited test -> (continuation of Deferred@641778548 after retry RPC@920173233) -> (continuation of Deferred@1402723071 after retry RPC@551943951) -> (continuation of Deferred@1627081167 after retry RPC@1869209432) -> (continuation of Deferred@599768419 after retry RPC@42808659) -> (continuation of Deferred@2134900209 after retry RPC@351500520))) after lookup
02:07:24.453 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] connecting to peer
02:07:24.454 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:24.454 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] cleaning up while in state CONNECTING due to: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:24.454 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 85635564f2c94c458f506c870a826b34(127.22.115.2:43855) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:25.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:25.284586 24452 raft_consensus.cc:493] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 85635564f2c94c458f506c870a826b34)
02:07:25.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:25.284688 24452 raft_consensus.cc:515] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:25.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:25.284863 24452 leader_election.cc:290] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 85635564f2c94c458f506c870a826b34 (127.22.115.2:43855), 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855)
02:07:25.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:25.285665 24144 leader_election.cc:336] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 85635564f2c94c458f506c870a826b34 (127.22.115.2:43855): Network error: Client connection negotiation failed: client connection to 127.22.115.2:43855: connect: Connection refused (error 111)
02:07:25.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:25.288264 24393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76ab850ff70d4ccca2590ffea11986ed" candidate_uuid: "619c08b8f3e246edabc0e4e315cf9201" candidate_term: 7 candidate_status { last_received { term: 6 index: 20 } } ignore_live_leader: false dest_uuid: "174330aa0d3e49c69cab90b20e4f7a56" is_pre_election: true
02:07:25.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:25.288398 24393 raft_consensus.cc:2468] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 619c08b8f3e246edabc0e4e315cf9201 in term 5.
02:07:25.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:25.288622 24141 leader_election.cc:304] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 174330aa0d3e49c69cab90b20e4f7a56, 619c08b8f3e246edabc0e4e315cf9201; no voters: 85635564f2c94c458f506c870a826b34
02:07:25.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:25.288731 24452 raft_consensus.cc:2804] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 6 FOLLOWER]: Leader pre-election won for term 7
02:07:25.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:25.288770 24452 raft_consensus.cc:493] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 85635564f2c94c458f506c870a826b34)
02:07:25.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:25.288792 24452 raft_consensus.cc:3060] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 6 FOLLOWER]: Advancing to term 7
02:07:25.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:25.289443 24452 raft_consensus.cc:515] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:25.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:25.289631 24452 leader_election.cc:290] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [CANDIDATE]: Term 7 election: Requested vote from peers 85635564f2c94c458f506c870a826b34 (127.22.115.2:43855), 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855)
02:07:25.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:25.289893 24393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76ab850ff70d4ccca2590ffea11986ed" candidate_uuid: "619c08b8f3e246edabc0e4e315cf9201" candidate_term: 7 candidate_status { last_received { term: 6 index: 20 } } ignore_live_leader: false dest_uuid: "174330aa0d3e49c69cab90b20e4f7a56"
02:07:25.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:25.289994 24393 raft_consensus.cc:3060] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 5 FOLLOWER]: Advancing to term 7
02:07:25.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:25.290191 24144 leader_election.cc:336] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 85635564f2c94c458f506c870a826b34 (127.22.115.2:43855): Network error: Client connection negotiation failed: client connection to 127.22.115.2:43855: connect: Connection refused (error 111)
02:07:25.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:25.290977 24393 raft_consensus.cc:2468] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 619c08b8f3e246edabc0e4e315cf9201 in term 7.
02:07:25.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:25.291163 24141 leader_election.cc:304] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 174330aa0d3e49c69cab90b20e4f7a56, 619c08b8f3e246edabc0e4e315cf9201; no voters: 85635564f2c94c458f506c870a826b34
02:07:25.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:25.291258 24452 raft_consensus.cc:2804] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 7 FOLLOWER]: Leader election won for term 7
02:07:25.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:25.291388 24452 raft_consensus.cc:697] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 7 LEADER]: Becoming Leader. State: Replica: 619c08b8f3e246edabc0e4e315cf9201, State: Running, Role: LEADER
02:07:25.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:25.291458 24452 consensus_queue.cc:237] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 6.20, Last appended by leader: 20, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:25.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:25.292179 23102 catalog_manager.cc:5696] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 reported cstate change: term changed from 6 to 7, leader changed from 85635564f2c94c458f506c870a826b34 (127.22.115.2) to 619c08b8f3e246edabc0e4e315cf9201 (127.22.115.1). New cstate: current_term: 7 leader_uuid: "619c08b8f3e246edabc0e4e315cf9201" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } health_report { overall_health: UNKNOWN } } }
02:07:25.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:25.824388 24393 raft_consensus.cc:1275] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 7 FOLLOWER]: Refusing update from remote peer 619c08b8f3e246edabc0e4e315cf9201: Log matching property violated. Preceding OpId in replica: term: 5 index: 16. Preceding OpId from leader: term: 7 index: 21. (index mismatch)
02:07:25.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:25.824802 24469 consensus_queue.cc:1048] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [LEADER]: Connected to new peer: Peer: permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 16, Time since last communication: 0.000s
02:07:25.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:25.828783 24144 consensus_peers.cc:597] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 -> Peer 85635564f2c94c458f506c870a826b34 (127.22.115.2:43855): Couldn't send request to peer 85635564f2c94c458f506c870a826b34. Status: Network error: Client connection negotiation failed: client connection to 127.22.115.2:43855: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:07:27.152 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:27.152 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855),619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)[L],85635564f2c94c458f506c870a826b34(127.22.115.2:43855)]}]
02:07:27.153 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=14, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=3237), Traces: [0ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [2ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [2ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [17ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [19ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [20ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [20ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [34ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [37ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [38ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [39ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [39ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [54ms] refreshing cache from master, [54ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [56ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [58ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [59ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [59ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [194ms] refreshing cache from master, [195ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [197ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [198ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [199ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [200ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [494ms] refreshing cache from master, [494ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [496ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [498ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [500ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [500ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [534ms] refreshing cache from master, [534ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [536ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [538ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [539ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [539ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [3234ms] refreshing cache from master, [3234ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [3237ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@33065829(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@5c026c68 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@641778548 after retry RPC@920173233) -> (continuation of Deferred@1402723071 after retry RPC@551943951) -> (continuation of Deferred@1627081167 after retry RPC@1869209432) -> (continuation of Deferred@599768419 after retry RPC@42808659) -> (continuation of Deferred@2134900209 after retry RPC@351500520) -> (continuation of Deferred@1621963821 after retry RPC@1872110824), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@13724f78 -> wakeup thread Time-limited test -> (continuation of Deferred@641778548 after retry RPC@920173233) -> (continuation of Deferred@1402723071 after retry RPC@551943951) -> (continuation of Deferred@1627081167 after retry RPC@1869209432) -> (continuation of Deferred@599768419 after retry RPC@42808659) -> (continuation of Deferred@2134900209 after retry RPC@351500520) -> (continuation of Deferred@1621963821 after retry RPC@1872110824))) after lookup
02:07:27.154 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] connecting to peer
02:07:27.155 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] Successfully connected to peer
02:07:27.173 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x1e1b64df, L:/127.0.0.1:39226 - R:/127.22.115.1:46047] using TOKEN/null
02:07:27.185 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.22.115.2:43855
02:07:27.187 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

02:07:27.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.188098 22988 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskTHXZc_/build/release/bin/kudu
02:07:27.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskTHXZc_/build/release/bin/kudu
02:07:27.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/wal
02:07:27.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/data
02:07:27.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:07:27.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:07:27.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:07:27.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:27.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:27.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:27.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/logs
02:07:27.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/data/info.pb
02:07:27.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:27.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:07:27.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:07:27.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:07:27.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:07:27.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:27.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:27.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:27.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:07:27.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:07:27.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.115.2:43855
02:07:27.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.115.2
02:07:27.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.115.62:38055,127.22.115.61:46503,127.22.115.60:37831
02:07:27.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37975
02:07:27.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.115.2
02:07:27.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.115.20:33307
02:07:27.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:27.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:27.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:07:27.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:27.273201 24493 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:07:27.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:27.273382 24493 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:07:27.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:27.273402 24493 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:07:27.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:27.274943 24493 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:07:27.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:27.275003 24493 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.115.2
02:07:27.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.276638 24493 tablet_server_runner.cc:78] Tablet server non-default flags:
02:07:27.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:27.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.115.20:33307
02:07:27.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:27.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:07:27.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/data
02:07:27.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/wal
02:07:27.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:27.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:27.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.115.2:43855
02:07:27.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:07:27.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:27.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:27.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/data/info.pb
02:07:27.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.115.2
02:07:27.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37975
02:07:27.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.115.62:38055,127.22.115.61:46503,127.22.115.60:37831
02:07:27.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:07:27.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.24493
02:07:27.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:27.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:07:27.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:07:27.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:27.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.115.2
02:07:27.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/logs
02:07:27.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:27.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:07:27.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:07:27.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:07:27.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:07:27.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
02:07:27.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
02:07:27.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 01:43:15 UTC on e7f111948823
02:07:27.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11949
02:07:27.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.276857 24493 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:07:27.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.277043 24493 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:07:27.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:27.279626 24500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:27.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:27.279631 24499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:27.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:27.279631 24502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:27.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.280045 24493 server_base.cc:1061] running on GCE node
02:07:27.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.280220 24493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:07:27.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.280407 24493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:07:27.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.281539 24493 hybrid_clock.cc:648] HybridClock initialized: now 1779070047281506 us; error 54 us; skew 500 ppm
02:07:27.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.282651 24493 webserver.cc:533] Webserver started at http://127.22.115.2:37975/ using document root <none> and password file <none>
02:07:27.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.282814 24493 fs_manager.cc:362] Metadata directory not provided
02:07:27.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.282853 24493 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:07:27.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.284006 24493 fs_manager.cc:714] Time spent opening directory manager: real 0.001s	user 0.001s	sys 0.000s
02:07:27.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.284662 24508 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:07:27.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.284842 24493 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.001s	sys 0.000s
02:07:27.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.284926 24493 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/data,/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/wal
02:07:27.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "85635564f2c94c458f506c870a826b34"
02:07:27.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:07:09 on dist-test-slave-37fl"
02:07:27.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.285228 24493 fs_report.cc:389] FS layout report
02:07:27.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:07:27.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/wal
02:07:27.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/wal
02:07:27.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/data/data
02:07:27.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:07:27.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:07:27.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:07:27.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:07:27.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:07:27.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:07:27.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:07:27.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:07:27.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:07:27.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:07:27.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:07:27.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.301707 24493 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:07:27.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.302086 24493 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:07:27.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.302222 24493 kserver.cc:163] Server-wide thread pool size limit: 3276
02:07:27.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.302459 24493 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:07:27.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.302986 24515 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
02:07:27.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.303776 24493 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:07:27.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.303833 24493 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s	user 0.000s	sys 0.000s
02:07:27.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.303869 24493 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:07:27.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.304495 24493 ts_tablet_manager.cc:616] Registered 1 tablets
02:07:27.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.304550 24493 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s	user 0.000s	sys 0.000s
02:07:27.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.304579 24515 tablet_bootstrap.cc:492] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34: Bootstrap starting.
02:07:27.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.311791 24493 rpc_server.cc:307] RPC server started. Bound to: 127.22.115.2:43855
02:07:27.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.311885 24622 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.115.2:43855 every 8 connection(s)
02:07:27.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.312269 24493 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/data/info.pb
02:07:27.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.312919 22988 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskTHXZc_/build/release/bin/kudu as pid 24493
02:07:27.313 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:07:27.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.314692 24515 log.cc:826] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34: Log is configured to *not* fsync() on all Append() calls
02:07:27.318 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 18 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563, tablet=76ab850ff70d4ccca2590ffea11986ed, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:07:27.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.319025 24515 tablet_bootstrap.cc:492] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=20 ignored=0} inserts{seen=15 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:07:27.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.319129 24623 heartbeater.cc:344] Connected to a master server at 127.22.115.62:38055
02:07:27.319 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.22.115.1:46047
02:07:27.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.319205 24623 heartbeater.cc:461] Registering TS with master...
02:07:27.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.319346 24515 tablet_bootstrap.cc:492] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34: Bootstrap complete.
02:07:27.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.319394 24623 heartbeater.cc:507] Master 127.22.115.62:38055 requested a full tablet report, sending...
02:07:27.319 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

02:07:27.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.319832 22988 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskTHXZc_/build/release/bin/kudu with pid 24126
02:07:27.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.320461 23028 ts_manager.cc:194] Re-registered known tserver with Master: 85635564f2c94c458f506c870a826b34 (127.22.115.2:43855)
02:07:27.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.321635 24625 heartbeater.cc:344] Connected to a master server at 127.22.115.61:46503
02:07:27.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.321693 24625 heartbeater.cc:461] Registering TS with master...
02:07:27.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.321797 24625 heartbeater.cc:507] Master 127.22.115.61:46503 requested a full tablet report, sending...
02:07:27.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.322091 23101 ts_manager.cc:194] Re-registered known tserver with Master: 85635564f2c94c458f506c870a826b34 (127.22.115.2:43855)
02:07:27.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.322500 23101 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.115.2:37693
02:07:27.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.323407 24515 ts_tablet_manager.cc:1403] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34: Time spent bootstrapping tablet: real 0.019s	user 0.009s	sys 0.005s
02:07:27.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.324699 24515 raft_consensus.cc:359] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:27.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.325033 24515 raft_consensus.cc:740] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85635564f2c94c458f506c870a826b34, State: Initialized, Role: FOLLOWER
02:07:27.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.325169 24515 consensus_queue.cc:260] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 20, Last appended: 6.20, Last appended by leader: 20, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:27.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.325467 24515 ts_tablet_manager.cc:1434] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
02:07:27.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.325778 24625 heartbeater.cc:499] Master 127.22.115.61:46503 was elected leader, sending a full tablet report...
02:07:27.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.327031 24624 heartbeater.cc:344] Connected to a master server at 127.22.115.60:37831
02:07:27.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.327102 24624 heartbeater.cc:461] Registering TS with master...
02:07:27.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.327203 24624 heartbeater.cc:507] Master 127.22.115.60:37831 requested a full tablet report, sending...
02:07:27.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:27.327517 23174 ts_manager.cc:194] Re-registered known tserver with Master: 85635564f2c94c458f506c870a826b34 (127.22.115.2:43855)
02:07:27.329 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:07:27.329 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] handling channelInactive
02:07:27.330 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] cleaning up while in state READY due to: connection closed
02:07:27.330 [DEBUG - Time-limited test] (Connection.java:748) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] connecting to peer
02:07:27.331 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:27.332 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] cleaning up while in state CONNECTING due to: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:27.332 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:27.352 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:27.353 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855),619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)[L],85635564f2c94c458f506c870a826b34(127.22.115.2:43855)]}]
02:07:27.353 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=2, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=23), Traces: [0ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [2ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [2ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [22ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@214181305(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@522be1ad -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@315bb171 -> wakeup thread Time-limited test)) after lookup
02:07:27.353 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] connecting to peer
02:07:27.354 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:27.355 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] cleaning up while in state CONNECTING due to: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:27.355 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:27.372 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:27.372 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855),619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)[L],85635564f2c94c458f506c870a826b34(127.22.115.2:43855)]}]
02:07:27.373 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=4, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=42), Traces: [0ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [2ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [2ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [22ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [23ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [25ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [25ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [42ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@214181305(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@522be1ad -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@903307079 after retry RPC@2126030426), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@315bb171 -> wakeup thread Time-limited test -> (continuation of Deferred@903307079 after retry RPC@2126030426))) after lookup
02:07:27.374 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] connecting to peer
02:07:27.375 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:27.375 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] cleaning up while in state CONNECTING due to: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:27.375 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:27.392 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:27.392 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855),619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)[L],85635564f2c94c458f506c870a826b34(127.22.115.2:43855)]}]
02:07:27.393 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=6, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=62), Traces: [0ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [2ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [2ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [22ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [23ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [25ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [25ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [42ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [43ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [45ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [45ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [59ms] refreshing cache from master, [59ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [62ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@214181305(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@522be1ad -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@903307079 after retry RPC@2126030426) -> (continuation of Deferred@2054714102 after retry RPC@1095340437), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@315bb171 -> wakeup thread Time-limited test -> (continuation of Deferred@903307079 after retry RPC@2126030426) -> (continuation of Deferred@2054714102 after retry RPC@1095340437))) after lookup
02:07:27.393 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] connecting to peer
02:07:27.394 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:27.394 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] cleaning up while in state CONNECTING due to: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:27.395 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:27.452 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:27.452 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855),619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)[L],85635564f2c94c458f506c870a826b34(127.22.115.2:43855)]}]
02:07:27.453 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=8, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=122), Traces: [0ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [2ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [2ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [22ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [23ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [25ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [25ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [42ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [43ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [45ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [45ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [59ms] refreshing cache from master, [59ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [62ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [63ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [65ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [65ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [119ms] refreshing cache from master, [119ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [122ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@214181305(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@522be1ad -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@903307079 after retry RPC@2126030426) -> (continuation of Deferred@2054714102 after retry RPC@1095340437) -> (continuation of Deferred@114727493 after retry RPC@186541619), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@315bb171 -> wakeup thread Time-limited test -> (continuation of Deferred@903307079 after retry RPC@2126030426) -> (continuation of Deferred@2054714102 after retry RPC@1095340437) -> (continuation of Deferred@114727493 after retry RPC@186541619))) after lookup
02:07:27.454 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] connecting to peer
02:07:27.455 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:27.455 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] cleaning up while in state CONNECTING due to: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:27.455 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:27.832 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:27.832 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855),619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)[L],85635564f2c94c458f506c870a826b34(127.22.115.2:43855)]}]
02:07:27.833 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=10, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=502), Traces: [0ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [2ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [2ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [22ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [23ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [25ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [25ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [42ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [43ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [45ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [45ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [59ms] refreshing cache from master, [59ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [62ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [63ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [65ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [65ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [119ms] refreshing cache from master, [119ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [122ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [123ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [125ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [125ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [500ms] refreshing cache from master, [500ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [502ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@214181305(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@522be1ad -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@903307079 after retry RPC@2126030426) -> (continuation of Deferred@2054714102 after retry RPC@1095340437) -> (continuation of Deferred@114727493 after retry RPC@186541619) -> (continuation of Deferred@1888143710 after retry RPC@140314242), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@315bb171 -> wakeup thread Time-limited test -> (continuation of Deferred@903307079 after retry RPC@2126030426) -> (continuation of Deferred@2054714102 after retry RPC@1095340437) -> (continuation of Deferred@114727493 after retry RPC@186541619) -> (continuation of Deferred@1888143710 after retry RPC@140314242))) after lookup
02:07:27.834 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] connecting to peer
02:07:27.834 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:27.835 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] cleaning up while in state CONNECTING due to: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:27.835 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:27.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:27.897686 24434 messenger.cc:194] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.22.115.3:43855
02:07:28.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:28.313053 24618 messenger.cc:194] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.22.115.2:43855
02:07:28.672 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:28.672 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855),619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)[L],85635564f2c94c458f506c870a826b34(127.22.115.2:43855)]}]
02:07:28.672 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=12, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=1342), Traces: [0ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [2ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [2ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [22ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [23ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [25ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [25ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [42ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [43ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [45ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [45ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [59ms] refreshing cache from master, [59ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [62ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [63ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [65ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [65ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [119ms] refreshing cache from master, [119ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [122ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [123ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [125ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [125ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [500ms] refreshing cache from master, [500ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [502ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [503ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [505ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [505ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [1339ms] refreshing cache from master, [1339ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [1341ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@214181305(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@522be1ad -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@903307079 after retry RPC@2126030426) -> (continuation of Deferred@2054714102 after retry RPC@1095340437) -> (continuation of Deferred@114727493 after retry RPC@186541619) -> (continuation of Deferred@1888143710 after retry RPC@140314242) -> (continuation of Deferred@1641525768 after retry RPC@22987174), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@315bb171 -> wakeup thread Time-limited test -> (continuation of Deferred@903307079 after retry RPC@2126030426) -> (continuation of Deferred@2054714102 after retry RPC@1095340437) -> (continuation of Deferred@114727493 after retry RPC@186541619) -> (continuation of Deferred@1888143710 after retry RPC@140314242) -> (continuation of Deferred@1641525768 after retry RPC@22987174))) after lookup
02:07:28.673 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] connecting to peer
02:07:28.674 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:28.674 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] cleaning up while in state CONNECTING due to: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:28.675 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:28.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:28.685406 24636 raft_consensus.cc:493] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 619c08b8f3e246edabc0e4e315cf9201)
02:07:28.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:28.685510 24636 raft_consensus.cc:515] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:28.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:28.685767 24636 leader_election.cc:290] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 619c08b8f3e246edabc0e4e315cf9201 (127.22.115.1:46047), 85635564f2c94c458f506c870a826b34 (127.22.115.2:43855)
02:07:28.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:28.686268 24328 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.115.1:46047: connect: Connection refused (error 111)
02:07:28.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:28.686761 24328 leader_election.cc:336] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 619c08b8f3e246edabc0e4e315cf9201 (127.22.115.1:46047): Network error: Client connection negotiation failed: client connection to 127.22.115.1:46047: connect: Connection refused (error 111)
02:07:28.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:28.689425 24577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76ab850ff70d4ccca2590ffea11986ed" candidate_uuid: "174330aa0d3e49c69cab90b20e4f7a56" candidate_term: 8 candidate_status { last_received { term: 7 index: 24 } } ignore_live_leader: false dest_uuid: "85635564f2c94c458f506c870a826b34" is_pre_election: true
02:07:28.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:28.689561 24577 raft_consensus.cc:2468] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 174330aa0d3e49c69cab90b20e4f7a56 in term 6.
02:07:28.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:28.689787 24328 leader_election.cc:304] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 174330aa0d3e49c69cab90b20e4f7a56, 85635564f2c94c458f506c870a826b34; no voters: 619c08b8f3e246edabc0e4e315cf9201
02:07:28.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:28.689960 24636 raft_consensus.cc:2804] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 7 FOLLOWER]: Leader pre-election won for term 8
02:07:28.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:28.690043 24636 raft_consensus.cc:493] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 619c08b8f3e246edabc0e4e315cf9201)
02:07:28.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:28.690075 24636 raft_consensus.cc:3060] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 7 FOLLOWER]: Advancing to term 8
02:07:28.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:28.690755 24636 raft_consensus.cc:515] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:28.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:28.690975 24636 leader_election.cc:290] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [CANDIDATE]: Term 8 election: Requested vote from peers 619c08b8f3e246edabc0e4e315cf9201 (127.22.115.1:46047), 85635564f2c94c458f506c870a826b34 (127.22.115.2:43855)
02:07:28.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:28.691298 24577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76ab850ff70d4ccca2590ffea11986ed" candidate_uuid: "174330aa0d3e49c69cab90b20e4f7a56" candidate_term: 8 candidate_status { last_received { term: 7 index: 24 } } ignore_live_leader: false dest_uuid: "85635564f2c94c458f506c870a826b34"
02:07:28.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:28.691390 24577 raft_consensus.cc:3060] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 6 FOLLOWER]: Advancing to term 8
02:07:28.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:28.691469 24328 leader_election.cc:336] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 619c08b8f3e246edabc0e4e315cf9201 (127.22.115.1:46047): Network error: Client connection negotiation failed: client connection to 127.22.115.1:46047: connect: Connection refused (error 111)
02:07:28.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:28.692466 24577 raft_consensus.cc:2468] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 174330aa0d3e49c69cab90b20e4f7a56 in term 8.
02:07:28.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:28.692677 24328 leader_election.cc:304] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 174330aa0d3e49c69cab90b20e4f7a56, 85635564f2c94c458f506c870a826b34; no voters: 619c08b8f3e246edabc0e4e315cf9201
02:07:28.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:28.692790 24636 raft_consensus.cc:2804] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 8 FOLLOWER]: Leader election won for term 8
02:07:28.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:28.692960 24636 raft_consensus.cc:697] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 8 LEADER]: Becoming Leader. State: Replica: 174330aa0d3e49c69cab90b20e4f7a56, State: Running, Role: LEADER
02:07:28.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:28.693077 24636 consensus_queue.cc:237] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 7.24, Last appended by leader: 24, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:28.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:28.693818 23101 catalog_manager.cc:5696] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 reported cstate change: term changed from 7 to 8, leader changed from 619c08b8f3e246edabc0e4e315cf9201 (127.22.115.1) to 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3). New cstate: current_term: 8 leader_uuid: "174330aa0d3e49c69cab90b20e4f7a56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } health_report { overall_health: HEALTHY } } }
02:07:29.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:29.174441 24577 raft_consensus.cc:1275] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 8 FOLLOWER]: Refusing update from remote peer 174330aa0d3e49c69cab90b20e4f7a56: Log matching property violated. Preceding OpId in replica: term: 6 index: 20. Preceding OpId from leader: term: 8 index: 25. (index mismatch)
02:07:29.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:29.174793 24636 consensus_queue.cc:1048] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 20, Time since last communication: 0.000s
02:07:29.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:29.177866 24328 consensus_peers.cc:597] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 -> Peer 619c08b8f3e246edabc0e4e315cf9201 (127.22.115.1:46047): Couldn't send request to peer 619c08b8f3e246edabc0e4e315cf9201. Status: Network error: Client connection negotiation failed: client connection to 127.22.115.1:46047: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:07:31.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:31.577834 24328 consensus_peers.cc:597] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 -> Peer 619c08b8f3e246edabc0e4e315cf9201 (127.22.115.1:46047): Couldn't send request to peer 619c08b8f3e246edabc0e4e315cf9201. Status: Network error: Client connection negotiation failed: client connection to 127.22.115.1:46047: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
02:07:31.752 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:31.753 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)[L],619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047),85635564f2c94c458f506c870a826b34(127.22.115.2:43855)]}]
02:07:31.753 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=14, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=4423), Traces: [0ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [2ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [2ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [22ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [23ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [25ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [25ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [40ms] refreshing cache from master, [40ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [42ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [43ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [45ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [45ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [59ms] refreshing cache from master, [59ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [62ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [63ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [65ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [65ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [119ms] refreshing cache from master, [119ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [122ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [123ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [125ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [125ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [500ms] refreshing cache from master, [500ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [502ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [503ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [505ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [505ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [1339ms] refreshing cache from master, [1339ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [1341ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [1343ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [1344ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [1345ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [4420ms] refreshing cache from master, [4420ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [4422ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@214181305(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@522be1ad -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@903307079 after retry RPC@2126030426) -> (continuation of Deferred@2054714102 after retry RPC@1095340437) -> (continuation of Deferred@114727493 after retry RPC@186541619) -> (continuation of Deferred@1888143710 after retry RPC@140314242) -> (continuation of Deferred@1641525768 after retry RPC@22987174) -> (continuation of Deferred@1746099888 after retry RPC@523756005), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@315bb171 -> wakeup thread Time-limited test -> (continuation of Deferred@903307079 after retry RPC@2126030426) -> (continuation of Deferred@2054714102 after retry RPC@1095340437) -> (continuation of Deferred@114727493 after retry RPC@186541619) -> (continuation of Deferred@1888143710 after retry RPC@140314242) -> (continuation of Deferred@1641525768 after retry RPC@22987174) -> (continuation of Deferred@1746099888 after retry RPC@523756005))) after lookup
02:07:31.754 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] connecting to peer
02:07:31.755 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] Successfully connected to peer
02:07:31.774 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0xa6782793, L:/127.0.0.1:58990 - R:/127.22.115.3:43855] using TOKEN/null
02:07:31.787 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.22.115.1:46047
02:07:31.787 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

02:07:31.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.788376 22988 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskTHXZc_/build/release/bin/kudu
02:07:31.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskTHXZc_/build/release/bin/kudu
02:07:31.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/wal
02:07:31.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/data
02:07:31.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:07:31.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:07:31.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:07:31.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:31.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:31.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:31.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/logs
02:07:31.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/data/info.pb
02:07:31.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:31.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:07:31.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:07:31.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:07:31.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:07:31.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:31.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:31.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:31.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:07:31.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:07:31.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.115.1:46047
02:07:31.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.115.1
02:07:31.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.115.62:38055,127.22.115.61:46503,127.22.115.60:37831
02:07:31.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=35073
02:07:31.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.115.1
02:07:31.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.115.20:33307
02:07:31.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:31.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:31.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:07:31.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:31.870020 24673 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:07:31.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:31.870223 24673 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:07:31.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:31.870246 24673 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:07:31.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:31.871794 24673 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:07:31.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:31.871853 24673 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.115.1
02:07:31.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.873513 24673 tablet_server_runner.cc:78] Tablet server non-default flags:
02:07:31.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:31.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.115.20:33307
02:07:31.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:31.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:07:31.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/data
02:07:31.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/wal
02:07:31.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:31.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:31.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.115.1:46047
02:07:31.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:07:31.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:31.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:31.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/data/info.pb
02:07:31.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.115.1
02:07:31.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=35073
02:07:31.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.115.62:38055,127.22.115.61:46503,127.22.115.60:37831
02:07:31.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:07:31.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.24673
02:07:31.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:31.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:07:31.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:07:31.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:31.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.115.1
02:07:31.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/logs
02:07:31.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:31.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:07:31.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:07:31.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:07:31.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:07:31.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
02:07:31.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
02:07:31.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 01:43:15 UTC on e7f111948823
02:07:31.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11949
02:07:31.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.873747 24673 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:07:31.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.874022 24673 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:07:31.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:31.876976 24679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:31.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:31.877229 24678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:31.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.877198 24673 server_base.cc:1061] running on GCE node
02:07:31.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:31.877298 24681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:31.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.877538 24673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:07:31.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.877740 24673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:07:31.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.878916 24673 hybrid_clock.cc:648] HybridClock initialized: now 1779070051878891 us; error 34 us; skew 500 ppm
02:07:31.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.880416 24673 webserver.cc:533] Webserver started at http://127.22.115.1:35073/ using document root <none> and password file <none>
02:07:31.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.880658 24673 fs_manager.cc:362] Metadata directory not provided
02:07:31.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.880780 24673 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:07:31.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.882162 24673 fs_manager.cc:714] Time spent opening directory manager: real 0.001s	user 0.001s	sys 0.000s
02:07:31.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.883069 24687 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:07:31.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.883327 24673 fs_manager.cc:730] Time spent opening block manager: real 0.001s	user 0.001s	sys 0.000s
02:07:31.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.883430 24673 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/data,/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/wal
02:07:31.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "619c08b8f3e246edabc0e4e315cf9201"
02:07:31.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:07:08 on dist-test-slave-37fl"
02:07:31.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.883762 24673 fs_report.cc:389] FS layout report
02:07:31.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:07:31.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/wal
02:07:31.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/wal
02:07:31.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/data/data
02:07:31.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:07:31.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:07:31.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:07:31.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:07:31.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:07:31.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:07:31.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:07:31.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:07:31.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:07:31.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:07:31.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:07:31.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.898113 24673 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:07:31.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.898476 24673 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:07:31.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.898610 24673 kserver.cc:163] Server-wide thread pool size limit: 3276
02:07:31.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.898881 24673 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:07:31.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.899647 24694 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
02:07:31.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.900995 24673 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:07:31.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.901074 24673 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.002s	user 0.000s	sys 0.000s
02:07:31.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.901166 24673 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:07:31.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.901762 24673 ts_tablet_manager.cc:616] Registered 1 tablets
02:07:31.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.901836 24673 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s	user 0.000s	sys 0.000s
02:07:31.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.901921 24694 tablet_bootstrap.cc:492] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201: Bootstrap starting.
02:07:31.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.910452 24673 rpc_server.cc:307] RPC server started. Bound to: 127.22.115.1:46047
02:07:31.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.910531 24801 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.115.1:46047 every 8 connection(s)
02:07:31.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.910892 24694 log.cc:826] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201: Log is configured to *not* fsync() on all Append() calls
02:07:31.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.910956 24673 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/data/info.pb
02:07:31.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.913566 22988 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskTHXZc_/build/release/bin/kudu as pid 24673
02:07:31.914 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:07:31.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.915460 24694 tablet_bootstrap.cc:492] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201: Bootstrap replayed 1/1 log segments. Stats: ops{read=24 overwritten=0 applied=24 ignored=0} inserts{seen=18 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:07:31.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.915789 24694 tablet_bootstrap.cc:492] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201: Bootstrap complete.
02:07:31.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.916967 24694 ts_tablet_manager.cc:1403] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201: Time spent bootstrapping tablet: real 0.015s	user 0.006s	sys 0.008s
02:07:31.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.918008 24804 heartbeater.cc:344] Connected to a master server at 127.22.115.60:37831
02:07:31.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.918126 24804 heartbeater.cc:461] Registering TS with master...
02:07:31.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.918362 24804 heartbeater.cc:507] Master 127.22.115.60:37831 requested a full tablet report, sending...
02:07:31.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.918159 24694 raft_consensus.cc:359] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:31.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.918429 24694 raft_consensus.cc:740] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 619c08b8f3e246edabc0e4e315cf9201, State: Initialized, Role: FOLLOWER
02:07:31.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.918553 24694 consensus_queue.cc:260] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 7.24, Last appended by leader: 24, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:31.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.918843 24694 ts_tablet_manager.cc:1434] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.002s
02:07:31.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.919209 23174 ts_manager.cc:194] Re-registered known tserver with Master: 619c08b8f3e246edabc0e4e315cf9201 (127.22.115.1:46047)
02:07:31.921 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 21 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563, tablet=76ab850ff70d4ccca2590ffea11986ed, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:07:31.922 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.22.115.3:43855
02:07:31.922 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

02:07:31.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.921343 24805 heartbeater.cc:344] Connected to a master server at 127.22.115.61:46503
02:07:31.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.921428 24805 heartbeater.cc:461] Registering TS with master...
02:07:31.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.921530 24805 heartbeater.cc:507] Master 127.22.115.61:46503 requested a full tablet report, sending...
02:07:31.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.921805 23101 ts_manager.cc:194] Re-registered known tserver with Master: 619c08b8f3e246edabc0e4e315cf9201 (127.22.115.1:46047)
02:07:31.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.922297 23101 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.115.1:45333
02:07:31.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.922667 24803 heartbeater.cc:344] Connected to a master server at 127.22.115.62:38055
02:07:31.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.922725 24803 heartbeater.cc:461] Registering TS with master...
02:07:31.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.922960 24805 heartbeater.cc:499] Master 127.22.115.61:46503 was elected leader, sending a full tablet report...
02:07:31.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.923025 24803 heartbeater.cc:507] Master 127.22.115.62:38055 requested a full tablet report, sending...
02:07:31.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.923199 22988 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskTHXZc_/build/release/bin/kudu with pid 24310
02:07:31.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:31.924829 23028 ts_manager.cc:194] Re-registered known tserver with Master: 619c08b8f3e246edabc0e4e315cf9201 (127.22.115.1:46047)
02:07:31.929 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] handling channelInactive
02:07:31.929 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] cleaning up while in state READY due to: connection closed
02:07:31.930 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:07:31.930 [DEBUG - Time-limited test] (Connection.java:748) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] connecting to peer
02:07:31.931 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:31.931 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] cleaning up while in state CONNECTING due to: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:31.932 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:31.952 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:31.953 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)[L],619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047),85635564f2c94c458f506c870a826b34(127.22.115.2:43855)]}]
02:07:31.953 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=2, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=23), Traces: [0ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [2ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [2ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [22ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@625093729(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@55ac9e5 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3fef878c -> wakeup thread Time-limited test)) after lookup
02:07:31.954 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] connecting to peer
02:07:31.954 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:31.955 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] cleaning up while in state CONNECTING due to: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:31.955 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:31.972 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:31.972 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)[L],619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047),85635564f2c94c458f506c870a826b34(127.22.115.2:43855)]}]
02:07:31.973 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=4, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=42), Traces: [0ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [2ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [2ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [22ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [23ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [25ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [25ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [42ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@625093729(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@55ac9e5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1752095581 after retry RPC@1620158635), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3fef878c -> wakeup thread Time-limited test -> (continuation of Deferred@1752095581 after retry RPC@1620158635))) after lookup
02:07:31.974 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] connecting to peer
02:07:31.975 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:31.975 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] cleaning up while in state CONNECTING due to: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:31.975 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:31.992 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:31.992 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)[L],619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047),85635564f2c94c458f506c870a826b34(127.22.115.2:43855)]}]
02:07:31.992 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=6, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=62), Traces: [0ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [2ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [2ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [22ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [23ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [25ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [25ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [42ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [43ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [45ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [45ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [59ms] refreshing cache from master, [59ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [61ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@625093729(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@55ac9e5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1752095581 after retry RPC@1620158635) -> (continuation of Deferred@1022699803 after retry RPC@1363942110), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3fef878c -> wakeup thread Time-limited test -> (continuation of Deferred@1752095581 after retry RPC@1620158635) -> (continuation of Deferred@1022699803 after retry RPC@1363942110))) after lookup
02:07:31.993 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] connecting to peer
02:07:31.994 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:31.994 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] cleaning up while in state CONNECTING due to: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:31.994 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:32.072 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:32.072 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)[L],619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047),85635564f2c94c458f506c870a826b34(127.22.115.2:43855)]}]
02:07:32.073 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=8, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=142), Traces: [0ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [2ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [2ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [22ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [23ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [25ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [25ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [42ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [43ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [45ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [45ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [59ms] refreshing cache from master, [59ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [61ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [62ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [64ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [64ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [139ms] refreshing cache from master, [140ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [142ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@625093729(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@55ac9e5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1752095581 after retry RPC@1620158635) -> (continuation of Deferred@1022699803 after retry RPC@1363942110) -> (continuation of Deferred@305245954 after retry RPC@588959088), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3fef878c -> wakeup thread Time-limited test -> (continuation of Deferred@1752095581 after retry RPC@1620158635) -> (continuation of Deferred@1022699803 after retry RPC@1363942110) -> (continuation of Deferred@305245954 after retry RPC@588959088))) after lookup
02:07:32.074 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] connecting to peer
02:07:32.075 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:32.075 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] cleaning up while in state CONNECTING due to: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:32.075 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:32.252 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:32.252 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)[L],619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047),85635564f2c94c458f506c870a826b34(127.22.115.2:43855)]}]
02:07:32.253 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=10, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=322), Traces: [0ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [2ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [2ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [22ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [23ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [25ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [25ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [42ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [43ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [45ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [45ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [59ms] refreshing cache from master, [59ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [61ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [62ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [64ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [64ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [139ms] refreshing cache from master, [140ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [142ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [143ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [145ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [145ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [319ms] refreshing cache from master, [320ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [322ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@625093729(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@55ac9e5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1752095581 after retry RPC@1620158635) -> (continuation of Deferred@1022699803 after retry RPC@1363942110) -> (continuation of Deferred@305245954 after retry RPC@588959088) -> (continuation of Deferred@1773645693 after retry RPC@473144950), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3fef878c -> wakeup thread Time-limited test -> (continuation of Deferred@1752095581 after retry RPC@1620158635) -> (continuation of Deferred@1022699803 after retry RPC@1363942110) -> (continuation of Deferred@305245954 after retry RPC@588959088) -> (continuation of Deferred@1773645693 after retry RPC@473144950))) after lookup
02:07:32.253 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] connecting to peer
02:07:32.254 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:32.254 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] cleaning up while in state CONNECTING due to: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:32.255 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:33.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:33.209698 24815 raft_consensus.cc:493] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
02:07:33.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:33.209883 24815 raft_consensus.cc:515] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:33.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:33.210182 24815 leader_election.cc:290] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 85635564f2c94c458f506c870a826b34 (127.22.115.2:43855), 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855)
02:07:33.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:33.210604 24688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.115.3:43855: connect: Connection refused (error 111)
02:07:33.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:33.211151 24688 leader_election.cc:336] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855): Network error: Client connection negotiation failed: client connection to 127.22.115.3:43855: connect: Connection refused (error 111)
02:07:33.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:33.213632 24577 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76ab850ff70d4ccca2590ffea11986ed" candidate_uuid: "619c08b8f3e246edabc0e4e315cf9201" candidate_term: 8 candidate_status { last_received { term: 7 index: 24 } } ignore_live_leader: false dest_uuid: "85635564f2c94c458f506c870a826b34" is_pre_election: true
02:07:33.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:33.214020 24691 leader_election.cc:304] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 619c08b8f3e246edabc0e4e315cf9201; no voters: 174330aa0d3e49c69cab90b20e4f7a56, 85635564f2c94c458f506c870a826b34
02:07:33.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:33.214205 24815 raft_consensus.cc:2749] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
02:07:33.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:33.287529 24820 raft_consensus.cc:493] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 174330aa0d3e49c69cab90b20e4f7a56)
02:07:33.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:33.287695 24820 raft_consensus.cc:515] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:33.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:33.288168 24820 leader_election.cc:290] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 619c08b8f3e246edabc0e4e315cf9201 (127.22.115.1:46047), 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855)
02:07:33.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:33.288676 24509 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.115.3:43855: connect: Connection refused (error 111)
02:07:33.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:33.289292 24509 leader_election.cc:336] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855): Network error: Client connection negotiation failed: client connection to 127.22.115.3:43855: connect: Connection refused (error 111)
02:07:33.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:33.291522 24756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76ab850ff70d4ccca2590ffea11986ed" candidate_uuid: "85635564f2c94c458f506c870a826b34" candidate_term: 9 candidate_status { last_received { term: 8 index: 28 } } ignore_live_leader: false dest_uuid: "619c08b8f3e246edabc0e4e315cf9201" is_pre_election: true
02:07:33.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:33.291654 24756 raft_consensus.cc:2468] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85635564f2c94c458f506c870a826b34 in term 7.
02:07:33.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:33.291973 24512 leader_election.cc:304] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 619c08b8f3e246edabc0e4e315cf9201, 85635564f2c94c458f506c870a826b34; no voters: 174330aa0d3e49c69cab90b20e4f7a56
02:07:33.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:33.292127 24820 raft_consensus.cc:2804] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 8 FOLLOWER]: Leader pre-election won for term 9
02:07:33.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:33.292198 24820 raft_consensus.cc:493] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 174330aa0d3e49c69cab90b20e4f7a56)
02:07:33.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:33.292228 24820 raft_consensus.cc:3060] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 8 FOLLOWER]: Advancing to term 9
02:07:33.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:33.292869 24820 raft_consensus.cc:515] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:33.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:33.293032 24820 leader_election.cc:290] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [CANDIDATE]: Term 9 election: Requested vote from peers 619c08b8f3e246edabc0e4e315cf9201 (127.22.115.1:46047), 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855)
02:07:33.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:33.293247 24756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76ab850ff70d4ccca2590ffea11986ed" candidate_uuid: "85635564f2c94c458f506c870a826b34" candidate_term: 9 candidate_status { last_received { term: 8 index: 28 } } ignore_live_leader: false dest_uuid: "619c08b8f3e246edabc0e4e315cf9201"
02:07:33.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:33.293313 24756 raft_consensus.cc:3060] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 7 FOLLOWER]: Advancing to term 9
02:07:33.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:33.293622 24509 leader_election.cc:336] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855): Network error: Client connection negotiation failed: client connection to 127.22.115.3:43855: connect: Connection refused (error 111)
02:07:33.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:33.294417 24756 raft_consensus.cc:2468] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85635564f2c94c458f506c870a826b34 in term 9.
02:07:33.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:33.294611 24512 leader_election.cc:304] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 619c08b8f3e246edabc0e4e315cf9201, 85635564f2c94c458f506c870a826b34; no voters: 174330aa0d3e49c69cab90b20e4f7a56
02:07:33.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:33.294747 24820 raft_consensus.cc:2804] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 9 FOLLOWER]: Leader election won for term 9
02:07:33.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:33.294916 24820 raft_consensus.cc:697] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 9 LEADER]: Becoming Leader. State: Replica: 85635564f2c94c458f506c870a826b34, State: Running, Role: LEADER
02:07:33.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:33.295007 24820 consensus_queue.cc:237] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 8.28, Last appended by leader: 28, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:33.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:33.296000 23101 catalog_manager.cc:5696] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 reported cstate change: term changed from 8 to 9, leader changed from 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3) to 85635564f2c94c458f506c870a826b34 (127.22.115.2). New cstate: current_term: 9 leader_uuid: "85635564f2c94c458f506c870a826b34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } health_report { overall_health: UNKNOWN } } }
02:07:33.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:33.733803 24756 raft_consensus.cc:1275] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 9 FOLLOWER]: Refusing update from remote peer 85635564f2c94c458f506c870a826b34: Log matching property violated. Preceding OpId in replica: term: 7 index: 24. Preceding OpId from leader: term: 9 index: 29. (index mismatch)
02:07:33.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:33.734351 24820 consensus_queue.cc:1048] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 24, Time since last communication: 0.000s
02:07:33.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:33.737818 24509 consensus_peers.cc:597] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 -> Peer 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855): Couldn't send request to peer 174330aa0d3e49c69cab90b20e4f7a56. Status: Network error: Client connection negotiation failed: client connection to 127.22.115.3:43855: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:07:33.972 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:33.972 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855),619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047),85635564f2c94c458f506c870a826b34(127.22.115.2:43855)[L]]}]
02:07:33.973 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=12, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=2042), Traces: [0ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [2ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [2ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [20ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [22ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [23ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [25ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [25ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [42ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [43ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [45ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [45ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [59ms] refreshing cache from master, [59ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [61ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [62ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [64ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [64ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [139ms] refreshing cache from master, [140ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [142ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [143ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [145ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [145ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [319ms] refreshing cache from master, [320ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [322ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [323ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [325ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [325ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [2039ms] refreshing cache from master, [2039ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [2042ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@625093729(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@55ac9e5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1752095581 after retry RPC@1620158635) -> (continuation of Deferred@1022699803 after retry RPC@1363942110) -> (continuation of Deferred@305245954 after retry RPC@588959088) -> (continuation of Deferred@1773645693 after retry RPC@473144950) -> (continuation of Deferred@1225534126 after retry RPC@1238158807), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3fef878c -> wakeup thread Time-limited test -> (continuation of Deferred@1752095581 after retry RPC@1620158635) -> (continuation of Deferred@1022699803 after retry RPC@1363942110) -> (continuation of Deferred@305245954 after retry RPC@588959088) -> (continuation of Deferred@1773645693 after retry RPC@473144950) -> (continuation of Deferred@1225534126 after retry RPC@1238158807))) after lookup
02:07:33.974 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] connecting to peer
02:07:33.975 [DEBUG - kudu-nio-7] (Connection.java:757) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] Successfully connected to peer
02:07:33.997 [DEBUG - kudu-nio-7] (Negotiator.java:919) Authenticated connection [id: 0x221587b7, L:/127.0.0.1:33812 - R:/127.22.115.2:43855] using TOKEN/null
02:07:34.006 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.22.115.3:43855
02:07:34.006 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

02:07:34.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.007166 22988 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskTHXZc_/build/release/bin/kudu
02:07:34.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskTHXZc_/build/release/bin/kudu
02:07:34.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/wal
02:07:34.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/data
02:07:34.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:07:34.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:07:34.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:07:34.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:34.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:34.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:34.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/logs
02:07:34.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/data/info.pb
02:07:34.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:34.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:07:34.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:07:34.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:07:34.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:07:34.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:34.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:34.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:34.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:07:34.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:07:34.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.115.3:43855
02:07:34.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.115.3
02:07:34.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.115.62:38055,127.22.115.61:46503,127.22.115.60:37831
02:07:34.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45263
02:07:34.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.115.3
02:07:34.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.115.20:33307
02:07:34.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:34.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:34.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:07:34.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:34.087517 24847 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:07:34.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:34.087749 24847 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:07:34.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:34.087790 24847 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:07:34.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:34.089654 24847 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:07:34.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:34.089716 24847 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.115.3
02:07:34.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.091378 24847 tablet_server_runner.cc:78] Tablet server non-default flags:
02:07:34.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:34.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.115.20:33307
02:07:34.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:34.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:07:34.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/data
02:07:34.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/wal
02:07:34.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:34.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:34.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.115.3:43855
02:07:34.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:07:34.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:34.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:34.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/data/info.pb
02:07:34.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.115.3
02:07:34.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45263
02:07:34.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.115.62:38055,127.22.115.61:46503,127.22.115.60:37831
02:07:34.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:07:34.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.24847
02:07:34.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:34.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:07:34.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:07:34.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:34.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.115.3
02:07:34.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/logs
02:07:34.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:34.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:07:34.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:07:34.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:07:34.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:07:34.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
02:07:34.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
02:07:34.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 01:43:15 UTC on e7f111948823
02:07:34.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11949
02:07:34.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.091634 24847 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:07:34.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.091851 24847 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:07:34.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:34.094851 24855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:34.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:34.094859 24853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:34.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:34.094830 24852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:34.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.095090 24847 server_base.cc:1061] running on GCE node
02:07:34.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.095280 24847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:07:34.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.095486 24847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:07:34.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.096647 24847 hybrid_clock.cc:648] HybridClock initialized: now 1779070054096631 us; error 83 us; skew 500 ppm
02:07:34.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.098135 24847 webserver.cc:533] Webserver started at http://127.22.115.3:45263/ using document root <none> and password file <none>
02:07:34.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.098313 24847 fs_manager.cc:362] Metadata directory not provided
02:07:34.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.098349 24847 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:07:34.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.099483 24847 fs_manager.cc:714] Time spent opening directory manager: real 0.001s	user 0.001s	sys 0.000s
02:07:34.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.100135 24861 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:07:34.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.100284 24847 fs_manager.cc:730] Time spent opening block manager: real 0.001s	user 0.000s	sys 0.000s
02:07:34.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.100345 24847 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/data,/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/wal
02:07:34.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "174330aa0d3e49c69cab90b20e4f7a56"
02:07:34.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:07:09 on dist-test-slave-37fl"
02:07:34.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.100600 24847 fs_report.cc:389] FS layout report
02:07:34.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:07:34.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/wal
02:07:34.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/wal
02:07:34.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/data/data
02:07:34.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:07:34.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:07:34.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:07:34.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:07:34.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:07:34.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:07:34.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:07:34.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:07:34.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:07:34.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:07:34.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:07:34.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.122108 24847 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:07:34.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.122390 24847 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:07:34.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.122479 24847 kserver.cc:163] Server-wide thread pool size limit: 3276
02:07:34.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.122653 24847 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:07:34.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.123060 24868 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
02:07:34.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.124096 24847 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:07:34.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.124158 24847 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s	user 0.000s	sys 0.000s
02:07:34.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.124185 24847 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:07:34.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.124742 24847 ts_tablet_manager.cc:616] Registered 1 tablets
02:07:34.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.124774 24847 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s	user 0.000s	sys 0.000s
02:07:34.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.124904 24868 tablet_bootstrap.cc:492] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56: Bootstrap starting.
02:07:34.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.132359 24847 rpc_server.cc:307] RPC server started. Bound to: 127.22.115.3:43855
02:07:34.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.132395 24975 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.115.3:43855 every 8 connection(s)
02:07:34.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.132725 24847 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/data/info.pb
02:07:34.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.135972 24868 log.cc:826] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56: Log is configured to *not* fsync() on all Append() calls
02:07:34.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.139532 24868 tablet_bootstrap.cc:492] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=21 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:07:34.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.139874 24868 tablet_bootstrap.cc:492] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56: Bootstrap complete.
02:07:34.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.140815 24868 ts_tablet_manager.cc:1403] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56: Time spent bootstrapping tablet: real 0.016s	user 0.011s	sys 0.004s
02:07:34.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.141727 24868 raft_consensus.cc:359] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:34.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.141961 24868 raft_consensus.cc:740] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 174330aa0d3e49c69cab90b20e4f7a56, State: Initialized, Role: FOLLOWER
02:07:34.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.142071 24868 consensus_queue.cc:260] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 8.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:34.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.142357 24868 ts_tablet_manager.cc:1434] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.002s
02:07:34.142 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:07:34.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.142464 22988 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskTHXZc_/build/release/bin/kudu as pid 24847
02:07:34.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.143905 24978 heartbeater.cc:344] Connected to a master server at 127.22.115.61:46503
02:07:34.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.144042 24978 heartbeater.cc:461] Registering TS with master...
02:07:34.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.144264 24978 heartbeater.cc:507] Master 127.22.115.61:46503 requested a full tablet report, sending...
02:07:34.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.144472 24976 heartbeater.cc:344] Connected to a master server at 127.22.115.62:38055
02:07:34.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.144539 24976 heartbeater.cc:461] Registering TS with master...
02:07:34.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.144630 24976 heartbeater.cc:507] Master 127.22.115.62:38055 requested a full tablet report, sending...
02:07:34.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.145004 24977 heartbeater.cc:344] Connected to a master server at 127.22.115.60:37831
02:07:34.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.145205 24977 heartbeater.cc:461] Registering TS with master...
02:07:34.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.145233 23028 ts_manager.cc:194] Re-registered known tserver with Master: 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855)
02:07:34.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.145491 24977 heartbeater.cc:507] Master 127.22.115.60:37831 requested a full tablet report, sending...
02:07:34.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.145524 23101 ts_manager.cc:194] Re-registered known tserver with Master: 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855)
02:07:34.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.146047 23101 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.115.3:47923
02:07:34.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.146548 23173 ts_manager.cc:194] Re-registered known tserver with Master: 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855)
02:07:34.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.147023 24978 heartbeater.cc:499] Master 127.22.115.61:46503 was elected leader, sending a full tablet report...
02:07:34.148 [DEBUG - kudu-nio-7] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 24 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563, tablet=76ab850ff70d4ccca2590ffea11986ed, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:07:34.149 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.22.115.2:43855
02:07:34.149 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

02:07:34.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:34.150182 22988 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskTHXZc_/build/release/bin/kudu with pid 24493
02:07:34.156 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:07:34.156 [DEBUG - kudu-nio-7] (Connection.java:239) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] handling channelInactive
02:07:34.156 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] cleaning up while in state READY due to: connection closed
02:07:34.157 [DEBUG - Time-limited test] (Connection.java:748) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] connecting to peer
02:07:34.157 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:34.158 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] cleaning up while in state CONNECTING due to: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:34.158 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 85635564f2c94c458f506c870a826b34(127.22.115.2:43855) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:34.172 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:34.172 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855),619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047),85635564f2c94c458f506c870a826b34(127.22.115.2:43855)[L]]}]
02:07:34.172 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=2, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=16), Traces: [0ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [2ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [2ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [13ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [16ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@845146906(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@1dd23af3 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@24beec84 -> wakeup thread Time-limited test)) after lookup
02:07:34.173 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] connecting to peer
02:07:34.174 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:34.174 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] cleaning up while in state CONNECTING due to: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:34.175 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 85635564f2c94c458f506c870a826b34(127.22.115.2:43855) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:34.191 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:34.192 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855),619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047),85635564f2c94c458f506c870a826b34(127.22.115.2:43855)[L]]}]
02:07:34.192 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=4, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=36), Traces: [0ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [2ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [2ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [13ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [16ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [17ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [19ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [19ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [35ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@845146906(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@1dd23af3 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@149142210 after retry RPC@1533722006), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@24beec84 -> wakeup thread Time-limited test -> (continuation of Deferred@149142210 after retry RPC@1533722006))) after lookup
02:07:34.193 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] connecting to peer
02:07:34.194 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:34.194 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] cleaning up while in state CONNECTING due to: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:34.194 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 85635564f2c94c458f506c870a826b34(127.22.115.2:43855) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:34.212 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:34.213 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855),619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047),85635564f2c94c458f506c870a826b34(127.22.115.2:43855)[L]]}]
02:07:34.213 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=6, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=57), Traces: [0ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [2ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [2ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [13ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [16ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [17ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [19ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [19ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [35ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [37ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [38ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [38ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [54ms] refreshing cache from master, [54ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [56ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@845146906(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@1dd23af3 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@149142210 after retry RPC@1533722006) -> (continuation of Deferred@1514935933 after retry RPC@1004618300), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@24beec84 -> wakeup thread Time-limited test -> (continuation of Deferred@149142210 after retry RPC@1533722006) -> (continuation of Deferred@1514935933 after retry RPC@1004618300))) after lookup
02:07:34.214 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] connecting to peer
02:07:34.215 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:34.215 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] cleaning up while in state CONNECTING due to: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:34.215 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 85635564f2c94c458f506c870a826b34(127.22.115.2:43855) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:34.273 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:34.273 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855),619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047),85635564f2c94c458f506c870a826b34(127.22.115.2:43855)[L]]}]
02:07:34.273 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=8, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=117), Traces: [0ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [2ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [2ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [13ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [16ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [17ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [19ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [19ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [35ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [37ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [38ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [38ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [54ms] refreshing cache from master, [54ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [56ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [57ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [59ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [59ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [114ms] refreshing cache from master, [114ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [116ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@845146906(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@1dd23af3 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@149142210 after retry RPC@1533722006) -> (continuation of Deferred@1514935933 after retry RPC@1004618300) -> (continuation of Deferred@1265122490 after retry RPC@2049533769), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@24beec84 -> wakeup thread Time-limited test -> (continuation of Deferred@149142210 after retry RPC@1533722006) -> (continuation of Deferred@1514935933 after retry RPC@1004618300) -> (continuation of Deferred@1265122490 after retry RPC@2049533769))) after lookup
02:07:34.274 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] connecting to peer
02:07:34.276 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:34.276 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] cleaning up while in state CONNECTING due to: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:34.276 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 85635564f2c94c458f506c870a826b34(127.22.115.2:43855) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:34.592 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:34.593 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855),619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047),85635564f2c94c458f506c870a826b34(127.22.115.2:43855)[L]]}]
02:07:34.593 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=10, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=437), Traces: [0ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [2ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [2ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [13ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [16ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [17ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [19ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [19ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [35ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [37ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [38ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [38ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [54ms] refreshing cache from master, [54ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [56ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [57ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [59ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [59ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [114ms] refreshing cache from master, [114ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [116ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [118ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [120ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [120ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [434ms] refreshing cache from master, [434ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [436ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@845146906(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@1dd23af3 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@149142210 after retry RPC@1533722006) -> (continuation of Deferred@1514935933 after retry RPC@1004618300) -> (continuation of Deferred@1265122490 after retry RPC@2049533769) -> (continuation of Deferred@756010236 after retry RPC@481162555), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@24beec84 -> wakeup thread Time-limited test -> (continuation of Deferred@149142210 after retry RPC@1533722006) -> (continuation of Deferred@1514935933 after retry RPC@1004618300) -> (continuation of Deferred@1265122490 after retry RPC@2049533769) -> (continuation of Deferred@756010236 after retry RPC@481162555))) after lookup
02:07:34.594 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] connecting to peer
02:07:34.595 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:34.595 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] cleaning up while in state CONNECTING due to: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:34.596 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 85635564f2c94c458f506c870a826b34(127.22.115.2:43855) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855
02:07:35.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:35.373769 24989 raft_consensus.cc:493] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
02:07:35.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:35.373967 24989 raft_consensus.cc:515] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:35.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:35.374229 24989 leader_election.cc:290] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 619c08b8f3e246edabc0e4e315cf9201 (127.22.115.1:46047), 85635564f2c94c458f506c870a826b34 (127.22.115.2:43855)
02:07:35.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:35.374994 24865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.115.2:43855: connect: Connection refused (error 111)
02:07:35.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:35.375519 24865 leader_election.cc:336] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 85635564f2c94c458f506c870a826b34 (127.22.115.2:43855): Network error: Client connection negotiation failed: client connection to 127.22.115.2:43855: connect: Connection refused (error 111)
02:07:35.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:35.377615 24756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76ab850ff70d4ccca2590ffea11986ed" candidate_uuid: "174330aa0d3e49c69cab90b20e4f7a56" candidate_term: 9 candidate_status { last_received { term: 8 index: 28 } } ignore_live_leader: false dest_uuid: "619c08b8f3e246edabc0e4e315cf9201" is_pre_election: true
02:07:35.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:35.377993 24865 leader_election.cc:304] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 174330aa0d3e49c69cab90b20e4f7a56; no voters: 619c08b8f3e246edabc0e4e315cf9201, 85635564f2c94c458f506c870a826b34
02:07:35.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:35.378149 24989 raft_consensus.cc:2749] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
02:07:35.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:35.506791 24994 raft_consensus.cc:493] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 85635564f2c94c458f506c870a826b34)
02:07:35.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:35.506906 24994 raft_consensus.cc:515] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:35.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:35.507069 24994 leader_election.cc:290] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 85635564f2c94c458f506c870a826b34 (127.22.115.2:43855), 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855)
02:07:35.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:35.508046 24691 leader_election.cc:336] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 85635564f2c94c458f506c870a826b34 (127.22.115.2:43855): Network error: Client connection negotiation failed: client connection to 127.22.115.2:43855: connect: Connection refused (error 111)
02:07:35.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:35.510522 24930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76ab850ff70d4ccca2590ffea11986ed" candidate_uuid: "619c08b8f3e246edabc0e4e315cf9201" candidate_term: 10 candidate_status { last_received { term: 9 index: 32 } } ignore_live_leader: false dest_uuid: "174330aa0d3e49c69cab90b20e4f7a56" is_pre_election: true
02:07:35.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:35.510653 24930 raft_consensus.cc:2468] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 619c08b8f3e246edabc0e4e315cf9201 in term 8.
02:07:35.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:35.510893 24688 leader_election.cc:304] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 174330aa0d3e49c69cab90b20e4f7a56, 619c08b8f3e246edabc0e4e315cf9201; no voters: 85635564f2c94c458f506c870a826b34
02:07:35.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:35.511034 24994 raft_consensus.cc:2804] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 9 FOLLOWER]: Leader pre-election won for term 10
02:07:35.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:35.511089 24994 raft_consensus.cc:493] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 85635564f2c94c458f506c870a826b34)
02:07:35.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:35.511116 24994 raft_consensus.cc:3060] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 9 FOLLOWER]: Advancing to term 10
02:07:35.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:35.511844 24994 raft_consensus.cc:515] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:35.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:35.512040 24994 leader_election.cc:290] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [CANDIDATE]: Term 10 election: Requested vote from peers 85635564f2c94c458f506c870a826b34 (127.22.115.2:43855), 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855)
02:07:35.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:35.512363 24930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76ab850ff70d4ccca2590ffea11986ed" candidate_uuid: "619c08b8f3e246edabc0e4e315cf9201" candidate_term: 10 candidate_status { last_received { term: 9 index: 32 } } ignore_live_leader: false dest_uuid: "174330aa0d3e49c69cab90b20e4f7a56"
02:07:35.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:35.512508 24930 raft_consensus.cc:3060] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 8 FOLLOWER]: Advancing to term 10
02:07:35.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:35.512722 24691 leader_election.cc:336] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 85635564f2c94c458f506c870a826b34 (127.22.115.2:43855): Network error: Client connection negotiation failed: client connection to 127.22.115.2:43855: connect: Connection refused (error 111)
02:07:35.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:35.513741 24930 raft_consensus.cc:2468] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 619c08b8f3e246edabc0e4e315cf9201 in term 10.
02:07:35.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:35.514016 24688 leader_election.cc:304] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 174330aa0d3e49c69cab90b20e4f7a56, 619c08b8f3e246edabc0e4e315cf9201; no voters: 85635564f2c94c458f506c870a826b34
02:07:35.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:35.514160 24994 raft_consensus.cc:2804] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 10 FOLLOWER]: Leader election won for term 10
02:07:35.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:35.514312 24994 raft_consensus.cc:697] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 10 LEADER]: Becoming Leader. State: Replica: 619c08b8f3e246edabc0e4e315cf9201, State: Running, Role: LEADER
02:07:35.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:35.514384 24994 consensus_queue.cc:237] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 9.32, Last appended by leader: 32, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:35.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:35.515237 23101 catalog_manager.cc:5696] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 reported cstate change: term changed from 9 to 10, leader changed from 85635564f2c94c458f506c870a826b34 (127.22.115.2) to 619c08b8f3e246edabc0e4e315cf9201 (127.22.115.1). New cstate: current_term: 10 leader_uuid: "619c08b8f3e246edabc0e4e315cf9201" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } health_report { overall_health: UNKNOWN } } }
02:07:36.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:36.046159 24691 consensus_peers.cc:597] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 -> Peer 85635564f2c94c458f506c870a826b34 (127.22.115.2:43855): Couldn't send request to peer 85635564f2c94c458f506c870a826b34. Status: Network error: Client connection negotiation failed: client connection to 127.22.115.2:43855: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:07:36.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.110122 24930 raft_consensus.cc:1275] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 10 FOLLOWER]: Refusing update from remote peer 619c08b8f3e246edabc0e4e315cf9201: Log matching property violated. Preceding OpId in replica: term: 8 index: 28. Preceding OpId from leader: term: 10 index: 33. (index mismatch)
02:07:36.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.110536 25011 consensus_queue.cc:1048] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [LEADER]: Connected to new peer: Peer: permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 28, Time since last communication: 0.000s
02:07:36.632 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:36.632 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855),619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)[L],85635564f2c94c458f506c870a826b34(127.22.115.2:43855)]}]
02:07:36.633 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=12, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=2476), Traces: [0ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [2ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [2ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [13ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [16ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [17ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [19ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [19ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [35ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [37ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [38ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [38ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [54ms] refreshing cache from master, [54ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [56ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [57ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [59ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [59ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [114ms] refreshing cache from master, [114ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [116ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [118ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [120ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [120ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [434ms] refreshing cache from master, [434ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [436ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [438ms] sending RPC to server 85635564f2c94c458f506c870a826b34, [440ms] received response from server 85635564f2c94c458f506c870a826b34: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [440ms] delaying RPC due to: Network error: Failed to connect to peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855): Connection refused: /127.22.115.2:43855, [2474ms] refreshing cache from master, [2474ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [2476ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@845146906(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@1dd23af3 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@149142210 after retry RPC@1533722006) -> (continuation of Deferred@1514935933 after retry RPC@1004618300) -> (continuation of Deferred@1265122490 after retry RPC@2049533769) -> (continuation of Deferred@756010236 after retry RPC@481162555) -> (continuation of Deferred@1019590584 after retry RPC@1328711990), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@24beec84 -> wakeup thread Time-limited test -> (continuation of Deferred@149142210 after retry RPC@1533722006) -> (continuation of Deferred@1514935933 after retry RPC@1004618300) -> (continuation of Deferred@1265122490 after retry RPC@2049533769) -> (continuation of Deferred@756010236 after retry RPC@481162555) -> (continuation of Deferred@1019590584 after retry RPC@1328711990))) after lookup
02:07:36.633 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] connecting to peer
02:07:36.634 [DEBUG - kudu-nio-6] (Connection.java:757) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] Successfully connected to peer
02:07:36.654 [DEBUG - kudu-nio-6] (Negotiator.java:919) Authenticated connection [id: 0x166fe152, L:/127.0.0.1:50112 - R:/127.22.115.1:46047] using TOKEN/null
02:07:36.663 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.22.115.2:43855
02:07:36.663 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

02:07:36.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.664108 22988 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskTHXZc_/build/release/bin/kudu
02:07:36.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskTHXZc_/build/release/bin/kudu
02:07:36.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/wal
02:07:36.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/data
02:07:36.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:07:36.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:07:36.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:07:36.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:36.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:36.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:36.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/logs
02:07:36.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/data/info.pb
02:07:36.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:36.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:07:36.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:07:36.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:07:36.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:07:36.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:36.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:36.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:36.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:07:36.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:07:36.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.115.2:43855
02:07:36.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.115.2
02:07:36.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.115.62:38055,127.22.115.61:46503,127.22.115.60:37831
02:07:36.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37975
02:07:36.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.115.2
02:07:36.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.115.20:33307
02:07:36.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:36.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:36.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:07:36.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:36.744809 25030 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:07:36.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:36.744990 25030 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:07:36.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:36.745010 25030 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:07:36.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:36.746448 25030 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:07:36.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:36.746500 25030 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.115.2
02:07:36.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.748016 25030 tablet_server_runner.cc:78] Tablet server non-default flags:
02:07:36.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:36.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.115.20:33307
02:07:36.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:36.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:07:36.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/data
02:07:36.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/wal
02:07:36.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:36.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:36.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.115.2:43855
02:07:36.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:07:36.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:36.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:36.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/data/info.pb
02:07:36.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.115.2
02:07:36.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37975
02:07:36.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.115.62:38055,127.22.115.61:46503,127.22.115.60:37831
02:07:36.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:07:36.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.25030
02:07:36.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:36.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:07:36.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:07:36.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:36.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.115.2
02:07:36.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/logs
02:07:36.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:36.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:07:36.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:07:36.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:07:36.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:07:36.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
02:07:36.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
02:07:36.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 01:43:15 UTC on e7f111948823
02:07:36.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11949
02:07:36.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.748217 25030 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:07:36.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.748389 25030 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:07:36.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:36.751034 25036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:36.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.751179 25030 server_base.cc:1061] running on GCE node
02:07:36.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:36.751250 25038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:36.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:36.751492 25035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:36.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.751709 25030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:07:36.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.752019 25030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:07:36.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.753201 25030 hybrid_clock.cc:648] HybridClock initialized: now 1779070056753174 us; error 36 us; skew 500 ppm
02:07:36.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.754474 25030 webserver.cc:533] Webserver started at http://127.22.115.2:37975/ using document root <none> and password file <none>
02:07:36.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.754710 25030 fs_manager.cc:362] Metadata directory not provided
02:07:36.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.754767 25030 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:07:36.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.756150 25030 fs_manager.cc:714] Time spent opening directory manager: real 0.001s	user 0.000s	sys 0.001s
02:07:36.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.756883 25044 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:07:36.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.757088 25030 fs_manager.cc:730] Time spent opening block manager: real 0.001s	user 0.000s	sys 0.001s
02:07:36.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.757186 25030 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/data,/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/wal
02:07:36.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "85635564f2c94c458f506c870a826b34"
02:07:36.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:07:09 on dist-test-slave-37fl"
02:07:36.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.757488 25030 fs_report.cc:389] FS layout report
02:07:36.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:07:36.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/wal
02:07:36.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/wal
02:07:36.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/data/data
02:07:36.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:07:36.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:07:36.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:07:36.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:07:36.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:07:36.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:07:36.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:07:36.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:07:36.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:07:36.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:07:36.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:07:36.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.784423 25030 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:07:36.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.784785 25030 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:07:36.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.784967 25030 kserver.cc:163] Server-wide thread pool size limit: 3276
02:07:36.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.785259 25030 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:07:36.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.785859 25051 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
02:07:36.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.786660 25030 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:07:36.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.786741 25030 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s	user 0.000s	sys 0.000s
02:07:36.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.786792 25030 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:07:36.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.787411 25030 ts_tablet_manager.cc:616] Registered 1 tablets
02:07:36.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.787470 25030 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s	user 0.000s	sys 0.000s
02:07:36.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.787550 25051 tablet_bootstrap.cc:492] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34: Bootstrap starting.
02:07:36.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.795015 25030 rpc_server.cc:307] RPC server started. Bound to: 127.22.115.2:43855
02:07:36.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.795056 25158 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.115.2:43855 every 8 connection(s)
02:07:36.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.795369 25030 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-1/data/info.pb
02:07:36.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.797901 25051 log.cc:826] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34: Log is configured to *not* fsync() on all Append() calls
02:07:36.800 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:07:36.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.799700 22988 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskTHXZc_/build/release/bin/kudu as pid 25030
02:07:36.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.802500 25160 heartbeater.cc:344] Connected to a master server at 127.22.115.60:37831
02:07:36.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.802591 25160 heartbeater.cc:461] Registering TS with master...
02:07:36.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.802767 25160 heartbeater.cc:507] Master 127.22.115.60:37831 requested a full tablet report, sending...
02:07:36.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.802942 25051 tablet_bootstrap.cc:492] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=24 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:07:36.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.803231 25051 tablet_bootstrap.cc:492] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34: Bootstrap complete.
02:07:36.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.803341 23173 ts_manager.cc:194] Re-registered known tserver with Master: 85635564f2c94c458f506c870a826b34 (127.22.115.2:43855)
02:07:36.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.804361 25051 ts_tablet_manager.cc:1403] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34: Time spent bootstrapping tablet: real 0.017s	user 0.009s	sys 0.007s
02:07:36.804 [DEBUG - kudu-nio-6] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 27 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563, tablet=76ab850ff70d4ccca2590ffea11986ed, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:07:36.805 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.22.115.1:46047
02:07:36.805 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

02:07:36.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.805660 25051 raft_consensus.cc:359] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 9 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:36.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.805910 22988 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskTHXZc_/build/release/bin/kudu with pid 24673
02:07:36.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.805949 25161 heartbeater.cc:344] Connected to a master server at 127.22.115.61:46503
02:07:36.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.806006 25161 heartbeater.cc:461] Registering TS with master...
02:07:36.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.806113 25161 heartbeater.cc:507] Master 127.22.115.61:46503 requested a full tablet report, sending...
02:07:36.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.806310 25159 heartbeater.cc:344] Connected to a master server at 127.22.115.62:38055
02:07:36.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.806342 25159 heartbeater.cc:461] Registering TS with master...
02:07:36.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.806416 25159 heartbeater.cc:507] Master 127.22.115.62:38055 requested a full tablet report, sending...
02:07:36.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.805960 25051 raft_consensus.cc:740] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85635564f2c94c458f506c870a826b34, State: Initialized, Role: FOLLOWER
02:07:36.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.806771 25051 consensus_queue.cc:260] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 9.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:36.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.807101 23102 ts_manager.cc:194] Re-registered known tserver with Master: 85635564f2c94c458f506c870a826b34 (127.22.115.2:43855)
02:07:36.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.807443 23102 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.115.2:51227
02:07:36.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.807693 23028 ts_manager.cc:194] Re-registered known tserver with Master: 85635564f2c94c458f506c870a826b34 (127.22.115.2:43855)
02:07:36.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.807879 25051 ts_tablet_manager.cc:1434] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
02:07:36.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:36.808154 25161 heartbeater.cc:499] Master 127.22.115.61:46503 was elected leader, sending a full tablet report...
02:07:36.813 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:07:36.813 [DEBUG - kudu-nio-6] (Connection.java:239) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] handling channelInactive
02:07:36.813 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] cleaning up while in state READY due to: connection closed
02:07:36.814 [DEBUG - Time-limited test] (Connection.java:748) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] connecting to peer
02:07:36.815 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:36.815 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] cleaning up while in state CONNECTING due to: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:36.815 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:36.832 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:36.832 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855),619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)[L],85635564f2c94c458f506c870a826b34(127.22.115.2:43855)]}]
02:07:36.833 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=2, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=19), Traces: [0ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [2ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [2ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [16ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [19ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@956863014(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@62a8c708 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@48603bf5 -> wakeup thread Time-limited test)) after lookup
02:07:36.834 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] connecting to peer
02:07:36.835 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:36.835 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] cleaning up while in state CONNECTING due to: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:36.835 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:36.851 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:36.852 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855),619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)[L],85635564f2c94c458f506c870a826b34(127.22.115.2:43855)]}]
02:07:36.852 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=4, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=38), Traces: [0ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [2ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [2ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [16ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [19ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [20ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [22ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [22ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [38ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@956863014(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@62a8c708 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@900859803 after retry RPC@2093231435), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@48603bf5 -> wakeup thread Time-limited test -> (continuation of Deferred@900859803 after retry RPC@2093231435))) after lookup
02:07:36.853 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] connecting to peer
02:07:36.854 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:36.854 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] cleaning up while in state CONNECTING due to: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:36.854 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:36.893 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:36.893 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855),619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)[L],85635564f2c94c458f506c870a826b34(127.22.115.2:43855)]}]
02:07:36.893 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=6, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=79), Traces: [0ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [2ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [2ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [16ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [19ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [20ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [22ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [22ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [38ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [39ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [41ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [41ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [79ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@956863014(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@62a8c708 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@900859803 after retry RPC@2093231435) -> (continuation of Deferred@90658255 after retry RPC@1477746269), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@48603bf5 -> wakeup thread Time-limited test -> (continuation of Deferred@900859803 after retry RPC@2093231435) -> (continuation of Deferred@90658255 after retry RPC@1477746269))) after lookup
02:07:36.894 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] connecting to peer
02:07:36.895 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:36.896 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] cleaning up while in state CONNECTING due to: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:36.896 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:36.953 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:36.953 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855),619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)[L],85635564f2c94c458f506c870a826b34(127.22.115.2:43855)]}]
02:07:36.954 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=8, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=140), Traces: [0ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [2ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [2ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [16ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [19ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [20ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [22ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [22ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [38ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [39ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [41ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [41ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [79ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [81ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [83ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [83ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [137ms] refreshing cache from master, [137ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [140ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@956863014(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@62a8c708 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@900859803 after retry RPC@2093231435) -> (continuation of Deferred@90658255 after retry RPC@1477746269) -> (continuation of Deferred@1049442474 after retry RPC@154475208), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@48603bf5 -> wakeup thread Time-limited test -> (continuation of Deferred@900859803 after retry RPC@2093231435) -> (continuation of Deferred@90658255 after retry RPC@1477746269) -> (continuation of Deferred@1049442474 after retry RPC@154475208))) after lookup
02:07:36.954 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] connecting to peer
02:07:36.955 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:36.956 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] cleaning up while in state CONNECTING due to: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:36.956 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:37.012 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:37.012 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855),619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)[L],85635564f2c94c458f506c870a826b34(127.22.115.2:43855)]}]
02:07:37.013 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=10, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=199), Traces: [0ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [2ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [2ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [16ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [19ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [20ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [22ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [22ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [38ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [39ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [41ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [41ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [79ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [81ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [83ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [83ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [137ms] refreshing cache from master, [137ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [140ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [141ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [143ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [143ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [197ms] refreshing cache from master, [197ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [199ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@956863014(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@62a8c708 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@900859803 after retry RPC@2093231435) -> (continuation of Deferred@90658255 after retry RPC@1477746269) -> (continuation of Deferred@1049442474 after retry RPC@154475208) -> (continuation of Deferred@1986948025 after retry RPC@1470946534), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@48603bf5 -> wakeup thread Time-limited test -> (continuation of Deferred@900859803 after retry RPC@2093231435) -> (continuation of Deferred@90658255 after retry RPC@1477746269) -> (continuation of Deferred@1049442474 after retry RPC@154475208) -> (continuation of Deferred@1986948025 after retry RPC@1470946534))) after lookup
02:07:37.013 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] connecting to peer
02:07:37.014 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:37.014 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] cleaning up while in state CONNECTING due to: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:37.014 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:37.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:37.133585 24971 messenger.cc:194] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.22.115.3:43855
02:07:37.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:37.796175 25154 messenger.cc:194] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.22.115.2:43855
02:07:38.152 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:38.152 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855),619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)[L],85635564f2c94c458f506c870a826b34(127.22.115.2:43855)]}]
02:07:38.152 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=12, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=1339), Traces: [0ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [2ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [2ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [16ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [19ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [20ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [22ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [22ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [38ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [39ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [41ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [41ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [79ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [81ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [83ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [83ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [137ms] refreshing cache from master, [137ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [140ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [141ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [143ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [143ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [197ms] refreshing cache from master, [197ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [199ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [200ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [201ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [201ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [1336ms] refreshing cache from master, [1336ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [1338ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@956863014(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@62a8c708 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@900859803 after retry RPC@2093231435) -> (continuation of Deferred@90658255 after retry RPC@1477746269) -> (continuation of Deferred@1049442474 after retry RPC@154475208) -> (continuation of Deferred@1986948025 after retry RPC@1470946534) -> (continuation of Deferred@761309964 after retry RPC@471468458), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@48603bf5 -> wakeup thread Time-limited test -> (continuation of Deferred@900859803 after retry RPC@2093231435) -> (continuation of Deferred@90658255 after retry RPC@1477746269) -> (continuation of Deferred@1049442474 after retry RPC@154475208) -> (continuation of Deferred@1986948025 after retry RPC@1470946534) -> (continuation of Deferred@761309964 after retry RPC@471468458))) after lookup
02:07:38.153 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] connecting to peer
02:07:38.154 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:38.154 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] cleaning up while in state CONNECTING due to: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:38.154 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047
02:07:38.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.163389 25172 raft_consensus.cc:493] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 619c08b8f3e246edabc0e4e315cf9201)
02:07:38.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.163493 25172 raft_consensus.cc:515] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:38.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.163668 25172 leader_election.cc:290] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 619c08b8f3e246edabc0e4e315cf9201 (127.22.115.1:46047), 85635564f2c94c458f506c870a826b34 (127.22.115.2:43855)
02:07:38.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:38.164599 24865 leader_election.cc:336] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 619c08b8f3e246edabc0e4e315cf9201 (127.22.115.1:46047): Network error: Client connection negotiation failed: client connection to 127.22.115.1:46047: connect: Connection refused (error 111)
02:07:38.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.167426 25113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76ab850ff70d4ccca2590ffea11986ed" candidate_uuid: "174330aa0d3e49c69cab90b20e4f7a56" candidate_term: 11 candidate_status { last_received { term: 10 index: 36 } } ignore_live_leader: false dest_uuid: "85635564f2c94c458f506c870a826b34" is_pre_election: true
02:07:38.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.167569 25113 raft_consensus.cc:2468] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 174330aa0d3e49c69cab90b20e4f7a56 in term 9.
02:07:38.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.167843 24865 leader_election.cc:304] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 174330aa0d3e49c69cab90b20e4f7a56, 85635564f2c94c458f506c870a826b34; no voters: 619c08b8f3e246edabc0e4e315cf9201
02:07:38.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.167991 25172 raft_consensus.cc:2804] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 10 FOLLOWER]: Leader pre-election won for term 11
02:07:38.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.168051 25172 raft_consensus.cc:493] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 619c08b8f3e246edabc0e4e315cf9201)
02:07:38.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.168105 25172 raft_consensus.cc:3060] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 10 FOLLOWER]: Advancing to term 11
02:07:38.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.168746 25172 raft_consensus.cc:515] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:38.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.168906 25172 leader_election.cc:290] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [CANDIDATE]: Term 11 election: Requested vote from peers 619c08b8f3e246edabc0e4e315cf9201 (127.22.115.1:46047), 85635564f2c94c458f506c870a826b34 (127.22.115.2:43855)
02:07:38.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.169279 25113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76ab850ff70d4ccca2590ffea11986ed" candidate_uuid: "174330aa0d3e49c69cab90b20e4f7a56" candidate_term: 11 candidate_status { last_received { term: 10 index: 36 } } ignore_live_leader: false dest_uuid: "85635564f2c94c458f506c870a826b34"
02:07:38.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.169399 25113 raft_consensus.cc:3060] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 9 FOLLOWER]: Advancing to term 11
02:07:38.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:38.169574 24865 leader_election.cc:336] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 619c08b8f3e246edabc0e4e315cf9201 (127.22.115.1:46047): Network error: Client connection negotiation failed: client connection to 127.22.115.1:46047: connect: Connection refused (error 111)
02:07:38.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.170444 25113 raft_consensus.cc:2468] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 174330aa0d3e49c69cab90b20e4f7a56 in term 11.
02:07:38.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.170650 24865 leader_election.cc:304] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 174330aa0d3e49c69cab90b20e4f7a56, 85635564f2c94c458f506c870a826b34; no voters: 619c08b8f3e246edabc0e4e315cf9201
02:07:38.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.170747 25172 raft_consensus.cc:2804] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 11 FOLLOWER]: Leader election won for term 11
02:07:38.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.170943 25172 raft_consensus.cc:697] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 11 LEADER]: Becoming Leader. State: Replica: 174330aa0d3e49c69cab90b20e4f7a56, State: Running, Role: LEADER
02:07:38.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.171077 25172 consensus_queue.cc:237] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 10.36, Last appended by leader: 36, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:38.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.171765 23102 catalog_manager.cc:5696] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 reported cstate change: term changed from 10 to 11, leader changed from 619c08b8f3e246edabc0e4e315cf9201 (127.22.115.1) to 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3). New cstate: current_term: 11 leader_uuid: "174330aa0d3e49c69cab90b20e4f7a56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } health_report { overall_health: HEALTHY } } }
02:07:38.412 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:38.413 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)[L],619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047),85635564f2c94c458f506c870a826b34(127.22.115.2:43855)]}]
02:07:38.413 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=14, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=1599), Traces: [0ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [2ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [2ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [16ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [19ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [20ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [22ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [22ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [38ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [39ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [41ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [41ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [79ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [81ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [83ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [83ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [137ms] refreshing cache from master, [137ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [140ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [141ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [143ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [143ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [197ms] refreshing cache from master, [197ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [199ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [200ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [201ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [201ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [1336ms] refreshing cache from master, [1336ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [1338ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [1340ms] sending RPC to server 619c08b8f3e246edabc0e4e315cf9201, [1341ms] received response from server 619c08b8f3e246edabc0e4e315cf9201: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [1341ms] delaying RPC due to: Network error: Failed to connect to peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047): Connection refused: /127.22.115.1:46047, [1596ms] refreshing cache from master, [1597ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [1599ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@956863014(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@62a8c708 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@900859803 after retry RPC@2093231435) -> (continuation of Deferred@90658255 after retry RPC@1477746269) -> (continuation of Deferred@1049442474 after retry RPC@154475208) -> (continuation of Deferred@1986948025 after retry RPC@1470946534) -> (continuation of Deferred@761309964 after retry RPC@471468458) -> (continuation of Deferred@484488201 after retry RPC@1350285670), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@48603bf5 -> wakeup thread Time-limited test -> (continuation of Deferred@900859803 after retry RPC@2093231435) -> (continuation of Deferred@90658255 after retry RPC@1477746269) -> (continuation of Deferred@1049442474 after retry RPC@154475208) -> (continuation of Deferred@1986948025 after retry RPC@1470946534) -> (continuation of Deferred@761309964 after retry RPC@471468458) -> (continuation of Deferred@484488201 after retry RPC@1350285670))) after lookup
02:07:38.414 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] connecting to peer
02:07:38.415 [DEBUG - kudu-nio-6] (Connection.java:757) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] Successfully connected to peer
02:07:38.430 [DEBUG - kudu-nio-6] (Negotiator.java:919) Authenticated connection [id: 0xd833b28d, L:/127.0.0.1:45944 - R:/127.22.115.3:43855] using TOKEN/null
02:07:38.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.432474 25113 raft_consensus.cc:1275] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 11 FOLLOWER]: Refusing update from remote peer 174330aa0d3e49c69cab90b20e4f7a56: Log matching property violated. Preceding OpId in replica: term: 9 index: 32. Preceding OpId from leader: term: 11 index: 38. (index mismatch)
02:07:38.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.432827 25177 consensus_queue.cc:1048] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 32, Time since last communication: 0.000s
02:07:38.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:38.433013 24865 consensus_peers.cc:597] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 -> Peer 619c08b8f3e246edabc0e4e315cf9201 (127.22.115.1:46047): Couldn't send request to peer 619c08b8f3e246edabc0e4e315cf9201. Status: Network error: Client connection negotiation failed: client connection to 127.22.115.1:46047: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:07:38.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.434247 25193 mvcc.cc:204] Tried to move back new op lower bound from 7287070952058843136 to 7287070947387682816. Current Snapshot: MvccSnapshot[applied={T|T < 7287070952058843136}]
02:07:38.444 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.22.115.1:46047
02:07:38.444 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

02:07:38.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.445214 22988 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskTHXZc_/build/release/bin/kudu
02:07:38.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskTHXZc_/build/release/bin/kudu
02:07:38.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/wal
02:07:38.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/data
02:07:38.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:07:38.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:07:38.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:07:38.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:38.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:38.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:38.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/logs
02:07:38.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/data/info.pb
02:07:38.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:38.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:07:38.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:07:38.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:07:38.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:07:38.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:38.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:38.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:38.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:07:38.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:07:38.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.115.1:46047
02:07:38.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.115.1
02:07:38.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.115.62:38055,127.22.115.61:46503,127.22.115.60:37831
02:07:38.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=35073
02:07:38.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.115.1
02:07:38.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.115.20:33307
02:07:38.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:38.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:38.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:07:38.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:38.524914 25199 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:07:38.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:38.525096 25199 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:07:38.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:38.525117 25199 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:07:38.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:38.526667 25199 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:07:38.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:38.526742 25199 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.115.1
02:07:38.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.528519 25199 tablet_server_runner.cc:78] Tablet server non-default flags:
02:07:38.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:38.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.115.20:33307
02:07:38.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:38.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:07:38.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/data
02:07:38.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/wal
02:07:38.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:38.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:38.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.115.1:46047
02:07:38.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:07:38.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:38.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:38.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/data/info.pb
02:07:38.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.115.1
02:07:38.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=35073
02:07:38.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.115.62:38055,127.22.115.61:46503,127.22.115.60:37831
02:07:38.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:07:38.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.25199
02:07:38.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:38.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:07:38.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:07:38.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:38.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.115.1
02:07:38.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/logs
02:07:38.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:38.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:07:38.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:07:38.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:07:38.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:07:38.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
02:07:38.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
02:07:38.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 01:43:15 UTC on e7f111948823
02:07:38.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11949
02:07:38.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.528726 25199 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:07:38.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.528908 25199 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:07:38.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:38.531782 25205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:38.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:38.531744 25204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:38.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:38.531772 25207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:38.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.532101 25199 server_base.cc:1061] running on GCE node
02:07:38.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.532624 25199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:07:38.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.532941 25199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:07:38.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.534135 25199 hybrid_clock.cc:648] HybridClock initialized: now 1779070058534083 us; error 63 us; skew 500 ppm
02:07:38.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.535509 25199 webserver.cc:533] Webserver started at http://127.22.115.1:35073/ using document root <none> and password file <none>
02:07:38.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.535718 25199 fs_manager.cc:362] Metadata directory not provided
02:07:38.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.535776 25199 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:07:38.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.537360 25199 fs_manager.cc:714] Time spent opening directory manager: real 0.001s	user 0.001s	sys 0.000s
02:07:38.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.538122 25213 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:07:38.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.538365 25199 fs_manager.cc:730] Time spent opening block manager: real 0.001s	user 0.001s	sys 0.000s
02:07:38.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.538463 25199 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/data,/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/wal
02:07:38.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "619c08b8f3e246edabc0e4e315cf9201"
02:07:38.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:07:08 on dist-test-slave-37fl"
02:07:38.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.538796 25199 fs_report.cc:389] FS layout report
02:07:38.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:07:38.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/wal
02:07:38.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/wal
02:07:38.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/data/data
02:07:38.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:07:38.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:07:38.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:07:38.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:07:38.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:07:38.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:07:38.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:07:38.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:07:38.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:07:38.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:07:38.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:07:38.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.561689 25199 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:07:38.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.562060 25199 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:07:38.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.562155 25199 kserver.cc:163] Server-wide thread pool size limit: 3276
02:07:38.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.562342 25199 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:07:38.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.562873 25220 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
02:07:38.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.564246 25199 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:07:38.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.564311 25199 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.002s	user 0.000s	sys 0.000s
02:07:38.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.564364 25199 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:07:38.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.565105 25199 ts_tablet_manager.cc:616] Registered 1 tablets
02:07:38.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.565182 25220 tablet_bootstrap.cc:492] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201: Bootstrap starting.
02:07:38.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.565191 25199 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s	user 0.001s	sys 0.000s
02:07:38.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.573881 25199 rpc_server.cc:307] RPC server started. Bound to: 127.22.115.1:46047
02:07:38.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.574391 25199 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-0/data/info.pb
02:07:38.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.577073 25220 log.cc:826] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201: Log is configured to *not* fsync() on all Append() calls
02:07:38.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.578742 25327 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.115.1:46047 every 8 connection(s)
02:07:38.582 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:07:38.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.581862 22988 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskTHXZc_/build/release/bin/kudu as pid 25199
02:07:38.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.583066 25220 tablet_bootstrap.cc:492] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201: Bootstrap replayed 1/1 log segments. Stats: ops{read=36 overwritten=0 applied=36 ignored=0} inserts{seen=27 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:07:38.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.583379 25220 tablet_bootstrap.cc:492] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201: Bootstrap complete.
02:07:38.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.584653 25220 ts_tablet_manager.cc:1403] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201: Time spent bootstrapping tablet: real 0.020s	user 0.013s	sys 0.005s
02:07:38.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.585585 25220 raft_consensus.cc:359] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 10 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:38.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.585768 25220 raft_consensus.cc:740] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 10 FOLLOWER]: Becoming Follower/Learner. State: Replica: 619c08b8f3e246edabc0e4e315cf9201, State: Initialized, Role: FOLLOWER
02:07:38.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.585829 25220 consensus_queue.cc:260] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 10.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:38.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.586110 25220 ts_tablet_manager.cc:1434] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.002s
02:07:38.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.590665 25329 heartbeater.cc:344] Connected to a master server at 127.22.115.60:37831
02:07:38.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.590775 25329 heartbeater.cc:461] Registering TS with master...
02:07:38.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.591001 25329 heartbeater.cc:507] Master 127.22.115.60:37831 requested a full tablet report, sending...
02:07:38.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.591022 25328 heartbeater.cc:344] Connected to a master server at 127.22.115.62:38055
02:07:38.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.591095 25328 heartbeater.cc:461] Registering TS with master...
02:07:38.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.591192 25328 heartbeater.cc:507] Master 127.22.115.62:38055 requested a full tablet report, sending...
02:07:38.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.592489 23173 ts_manager.cc:194] Re-registered known tserver with Master: 619c08b8f3e246edabc0e4e315cf9201 (127.22.115.1:46047)
02:07:38.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.592864 23029 ts_manager.cc:194] Re-registered known tserver with Master: 619c08b8f3e246edabc0e4e315cf9201 (127.22.115.1:46047)
02:07:38.593 [DEBUG - kudu-nio-6] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 30 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563, tablet=76ab850ff70d4ccca2590ffea11986ed, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:07:38.593 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.22.115.3:43855
02:07:38.594 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

02:07:38.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.594424 22988 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskTHXZc_/build/release/bin/kudu with pid 24847
02:07:38.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.600543 25330 heartbeater.cc:344] Connected to a master server at 127.22.115.61:46503
02:07:38.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.600663 25330 heartbeater.cc:461] Registering TS with master...
02:07:38.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.600813 25330 heartbeater.cc:507] Master 127.22.115.61:46503 requested a full tablet report, sending...
02:07:38.602 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:07:38.602 [DEBUG - kudu-nio-6] (Connection.java:239) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] handling channelInactive
02:07:38.602 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] cleaning up while in state READY due to: connection closed
02:07:38.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.601334 23102 ts_manager.cc:194] Re-registered known tserver with Master: 619c08b8f3e246edabc0e4e315cf9201 (127.22.115.1:46047)
02:07:38.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:38.601840 23102 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.115.1:51687
02:07:38.603 [DEBUG - Time-limited test] (Connection.java:748) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] connecting to peer
02:07:38.604 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:38.604 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] cleaning up while in state CONNECTING due to: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:38.604 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:38.612 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:38.613 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)[L],619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047),85635564f2c94c458f506c870a826b34(127.22.115.2:43855)]}]
02:07:38.613 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=2, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=10), Traces: [0ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [2ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [2ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [10ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@722790555(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@26a9c416 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@296ed13c -> wakeup thread Time-limited test)) after lookup
02:07:38.614 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] connecting to peer
02:07:38.615 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:38.615 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] cleaning up while in state CONNECTING due to: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:38.615 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:38.632 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:38.632 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)[L],619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047),85635564f2c94c458f506c870a826b34(127.22.115.2:43855)]}]
02:07:38.632 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=4, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=30), Traces: [0ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [2ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [2ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [10ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [11ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [13ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [13ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [27ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [30ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@722790555(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@26a9c416 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@366444046 after retry RPC@1257224889), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@296ed13c -> wakeup thread Time-limited test -> (continuation of Deferred@366444046 after retry RPC@1257224889))) after lookup
02:07:38.633 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] connecting to peer
02:07:38.634 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:38.634 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] cleaning up while in state CONNECTING due to: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:38.634 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:38.651 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:38.652 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)[L],619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047),85635564f2c94c458f506c870a826b34(127.22.115.2:43855)]}]
02:07:38.652 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=6, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=50), Traces: [0ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [2ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [2ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [10ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [11ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [13ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [13ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [27ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [30ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [31ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [32ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [33ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [49ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@722790555(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@26a9c416 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@366444046 after retry RPC@1257224889) -> (continuation of Deferred@186742159 after retry RPC@1009527840), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@296ed13c -> wakeup thread Time-limited test -> (continuation of Deferred@366444046 after retry RPC@1257224889) -> (continuation of Deferred@186742159 after retry RPC@1009527840))) after lookup
02:07:38.653 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] connecting to peer
02:07:38.654 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:38.654 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] cleaning up while in state CONNECTING due to: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:38.654 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:38.712 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:38.712 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)[L],619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047),85635564f2c94c458f506c870a826b34(127.22.115.2:43855)]}]
02:07:38.713 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=8, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=110), Traces: [0ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [2ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [2ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [10ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [11ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [13ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [13ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [27ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [30ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [31ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [32ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [33ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [49ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [51ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [52ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [52ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [107ms] refreshing cache from master, [107ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [110ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@722790555(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@26a9c416 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@366444046 after retry RPC@1257224889) -> (continuation of Deferred@186742159 after retry RPC@1009527840) -> (continuation of Deferred@204161429 after retry RPC@1885884370), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@296ed13c -> wakeup thread Time-limited test -> (continuation of Deferred@366444046 after retry RPC@1257224889) -> (continuation of Deferred@186742159 after retry RPC@1009527840) -> (continuation of Deferred@204161429 after retry RPC@1885884370))) after lookup
02:07:38.714 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] connecting to peer
02:07:38.715 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:38.715 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] cleaning up while in state CONNECTING due to: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:38.715 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:38.932 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:38.932 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)[L],619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047),85635564f2c94c458f506c870a826b34(127.22.115.2:43855)]}]
02:07:38.932 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=10, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=330), Traces: [0ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [2ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [2ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [10ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [11ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [13ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [13ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [27ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [30ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [31ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [32ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [33ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [49ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [51ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [52ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [52ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [107ms] refreshing cache from master, [107ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [110ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [112ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [113ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [113ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [327ms] refreshing cache from master, [327ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [329ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@722790555(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@26a9c416 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@366444046 after retry RPC@1257224889) -> (continuation of Deferred@186742159 after retry RPC@1009527840) -> (continuation of Deferred@204161429 after retry RPC@1885884370) -> (continuation of Deferred@2028339555 after retry RPC@1254279026), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@296ed13c -> wakeup thread Time-limited test -> (continuation of Deferred@366444046 after retry RPC@1257224889) -> (continuation of Deferred@186742159 after retry RPC@1009527840) -> (continuation of Deferred@204161429 after retry RPC@1885884370) -> (continuation of Deferred@2028339555 after retry RPC@1254279026))) after lookup
02:07:38.933 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] connecting to peer
02:07:38.934 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:38.934 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] cleaning up while in state CONNECTING due to: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:38.934 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855) for tablet 76ab850ff70d4ccca2590ffea11986ed: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855
02:07:39.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:39.603677 25330 heartbeater.cc:499] Master 127.22.115.61:46503 was elected leader, sending a full tablet report...
02:07:39.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:39.944626 25340 raft_consensus.cc:493] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 174330aa0d3e49c69cab90b20e4f7a56)
02:07:39.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:39.944753 25340 raft_consensus.cc:515] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:39.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:39.945087 25340 leader_election.cc:290] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 619c08b8f3e246edabc0e4e315cf9201 (127.22.115.1:46047), 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855)
02:07:39.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:39.945523 25045 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.115.3:43855: connect: Connection refused (error 111)
02:07:39.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:39.946015 25045 leader_election.cc:336] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855): Network error: Client connection negotiation failed: client connection to 127.22.115.3:43855: connect: Connection refused (error 111)
02:07:39.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:39.948397 25282 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76ab850ff70d4ccca2590ffea11986ed" candidate_uuid: "85635564f2c94c458f506c870a826b34" candidate_term: 12 candidate_status { last_received { term: 11 index: 40 } } ignore_live_leader: false dest_uuid: "619c08b8f3e246edabc0e4e315cf9201" is_pre_election: true
02:07:39.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:39.948531 25282 raft_consensus.cc:2468] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85635564f2c94c458f506c870a826b34 in term 10.
02:07:39.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:39.948767 25048 leader_election.cc:304] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 619c08b8f3e246edabc0e4e315cf9201, 85635564f2c94c458f506c870a826b34; no voters: 174330aa0d3e49c69cab90b20e4f7a56
02:07:39.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:39.948979 25340 raft_consensus.cc:2804] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 11 FOLLOWER]: Leader pre-election won for term 12
02:07:39.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:39.949038 25340 raft_consensus.cc:493] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 174330aa0d3e49c69cab90b20e4f7a56)
02:07:39.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:39.949052 25340 raft_consensus.cc:3060] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 11 FOLLOWER]: Advancing to term 12
02:07:39.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:39.949594 25340 raft_consensus.cc:515] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:39.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:39.949849 25340 leader_election.cc:290] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [CANDIDATE]: Term 12 election: Requested vote from peers 619c08b8f3e246edabc0e4e315cf9201 (127.22.115.1:46047), 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855)
02:07:39.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:39.950120 25282 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76ab850ff70d4ccca2590ffea11986ed" candidate_uuid: "85635564f2c94c458f506c870a826b34" candidate_term: 12 candidate_status { last_received { term: 11 index: 40 } } ignore_live_leader: false dest_uuid: "619c08b8f3e246edabc0e4e315cf9201"
02:07:39.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:39.950209 25282 raft_consensus.cc:3060] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 10 FOLLOWER]: Advancing to term 12
02:07:39.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:39.950620 25045 leader_election.cc:336] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855): Network error: Client connection negotiation failed: client connection to 127.22.115.3:43855: connect: Connection refused (error 111)
02:07:39.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:39.951267 25282 raft_consensus.cc:2468] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85635564f2c94c458f506c870a826b34 in term 12.
02:07:39.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:39.951470 25048 leader_election.cc:304] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 619c08b8f3e246edabc0e4e315cf9201, 85635564f2c94c458f506c870a826b34; no voters: 174330aa0d3e49c69cab90b20e4f7a56
02:07:39.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:39.951563 25340 raft_consensus.cc:2804] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 12 FOLLOWER]: Leader election won for term 12
02:07:39.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:39.951727 25340 raft_consensus.cc:697] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [term 12 LEADER]: Becoming Leader. State: Replica: 85635564f2c94c458f506c870a826b34, State: Running, Role: LEADER
02:07:39.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:39.951805 25340 consensus_queue.cc:237] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 11.40, Last appended by leader: 40, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:39.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:39.953056 23102 catalog_manager.cc:5696] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 reported cstate change: term changed from 11 to 12, leader changed from 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3) to 85635564f2c94c458f506c870a826b34 (127.22.115.2). New cstate: current_term: 12 leader_uuid: "85635564f2c94c458f506c870a826b34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } health_report { overall_health: UNKNOWN } } }
02:07:40.232 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 76ab850ff70d4ccca2590ffea11986ed for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563' with partition [<start>, <end>)
02:07:40.232 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=76ab850ff70d4ccca2590ffea11986ed@[174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855),619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047),85635564f2c94c458f506c870a826b34(127.22.115.2:43855)[L]]}]
02:07:40.232 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=76ab850ff70d4ccca2590ffea11986ed, attempt=12, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=1630), Traces: [0ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [2ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [2ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [10ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [11ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [13ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [13ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [27ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [30ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [31ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [32ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [33ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [47ms] refreshing cache from master, [47ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [49ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [51ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [52ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [52ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [107ms] refreshing cache from master, [107ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [110ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [112ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [113ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [113ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [327ms] refreshing cache from master, [327ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [329ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, [331ms] sending RPC to server 174330aa0d3e49c69cab90b20e4f7a56, [332ms] received response from server 174330aa0d3e49c69cab90b20e4f7a56: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [332ms] delaying RPC due to: Network error: Failed to connect to peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855): Connection refused: /127.22.115.3:43855, [1627ms] refreshing cache from master, [1627ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.115.61:46503, [1629ms] Sub RPC GetTableLocations: received response from server master-127.22.115.61:46503: OK, deferred=Deferred@722790555(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007f7e10336958@26a9c416 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@366444046 after retry RPC@1257224889) -> (continuation of Deferred@186742159 after retry RPC@1009527840) -> (continuation of Deferred@204161429 after retry RPC@1885884370) -> (continuation of Deferred@2028339555 after retry RPC@1254279026) -> (continuation of Deferred@1209224523 after retry RPC@347535980), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@296ed13c -> wakeup thread Time-limited test -> (continuation of Deferred@366444046 after retry RPC@1257224889) -> (continuation of Deferred@186742159 after retry RPC@1009527840) -> (continuation of Deferred@204161429 after retry RPC@1885884370) -> (continuation of Deferred@2028339555 after retry RPC@1254279026) -> (continuation of Deferred@1209224523 after retry RPC@347535980))) after lookup
02:07:40.233 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] connecting to peer
02:07:40.234 [DEBUG - kudu-nio-5] (Connection.java:757) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] Successfully connected to peer
02:07:40.261 [DEBUG - kudu-nio-5] (Negotiator.java:919) Authenticated connection [id: 0x338b1762, L:/127.0.0.1:45116 - R:/127.22.115.2:43855] using TOKEN/null
02:07:40.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.263517 25282 raft_consensus.cc:1275] T 76ab850ff70d4ccca2590ffea11986ed P 619c08b8f3e246edabc0e4e315cf9201 [term 12 FOLLOWER]: Refusing update from remote peer 85635564f2c94c458f506c870a826b34: Log matching property violated. Preceding OpId in replica: term: 10 index: 36. Preceding OpId from leader: term: 12 index: 42. (index mismatch)
02:07:40.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:40.263935 25045 consensus_peers.cc:597] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 -> Peer 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855): Couldn't send request to peer 174330aa0d3e49c69cab90b20e4f7a56. Status: Network error: Client connection negotiation failed: client connection to 127.22.115.3:43855: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:07:40.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.264055 25345 consensus_queue.cc:1048] T 76ab850ff70d4ccca2590ffea11986ed P 85635564f2c94c458f506c870a826b34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 36, Time since last communication: 0.000s
02:07:40.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.266145 25360 mvcc.cc:204] Tried to move back new op lower bound from 7287070959379853312 to 7287070958269267968. Current Snapshot: MvccSnapshot[applied={T|T < 7287070959379853312 or (T in {7287070959379853312})}]
02:07:40.273 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.22.115.3:43855
02:07:40.273 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

02:07:40.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.274168 22988 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskTHXZc_/build/release/bin/kudu
02:07:40.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskTHXZc_/build/release/bin/kudu
02:07:40.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/wal
02:07:40.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/data
02:07:40.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:07:40.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:07:40.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:07:40.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:40.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:40.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:40.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/logs
02:07:40.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/data/info.pb
02:07:40.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:40.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:07:40.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:07:40.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:07:40.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:07:40.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:40.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:40.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:40.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:07:40.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:07:40.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.115.3:43855
02:07:40.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.115.3
02:07:40.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.115.62:38055,127.22.115.61:46503,127.22.115.60:37831
02:07:40.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45263
02:07:40.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.115.3
02:07:40.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.115.20:33307
02:07:40.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:40.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:40.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:07:40.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:40.352031 25366 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:07:40.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:40.352187 25366 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:07:40.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:40.352207 25366 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:07:40.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:40.353585 25366 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:07:40.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:40.353631 25366 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.115.3
02:07:40.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.355044 25366 tablet_server_runner.cc:78] Tablet server non-default flags:
02:07:40.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:07:40.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.22.115.20:33307
02:07:40.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:07:40.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:07:40.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/data
02:07:40.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/wal
02:07:40.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:07:40.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:07:40.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.22.115.3:43855
02:07:40.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:07:40.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:07:40.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:07:40.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/data/info.pb
02:07:40.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.22.115.3
02:07:40.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45263
02:07:40.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.22.115.62:38055,127.22.115.61:46503,127.22.115.60:37831
02:07:40.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:07:40.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.25366
02:07:40.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:07:40.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:07:40.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:07:40.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:07:40.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.22.115.3
02:07:40.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/logs
02:07:40.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:07:40.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:07:40.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:07:40.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:07:40.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:07:40.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
02:07:40.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
02:07:40.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 01:43:15 UTC on e7f111948823
02:07:40.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11949
02:07:40.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.355224 25366 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:07:40.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.355386 25366 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:07:40.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:40.357985 25374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:40.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:40.357985 25371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:40.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:07:40.358019 25372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:07:40.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.358537 25366 server_base.cc:1061] running on GCE node
02:07:40.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.358686 25366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:07:40.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.358923 25366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:07:40.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.360025 25366 hybrid_clock.cc:648] HybridClock initialized: now 1779070060359984 us; error 48 us; skew 500 ppm
02:07:40.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.361402 25366 webserver.cc:533] Webserver started at http://127.22.115.3:45263/ using document root <none> and password file <none>
02:07:40.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.361629 25366 fs_manager.cc:362] Metadata directory not provided
02:07:40.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.361702 25366 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:07:40.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.363246 25366 fs_manager.cc:714] Time spent opening directory manager: real 0.001s	user 0.001s	sys 0.000s
02:07:40.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.364089 25380 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:07:40.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.364346 25366 fs_manager.cc:730] Time spent opening block manager: real 0.001s	user 0.001s	sys 0.000s
02:07:40.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.364423 25366 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/data,/tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/wal
02:07:40.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "174330aa0d3e49c69cab90b20e4f7a56"
02:07:40.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:07:09 on dist-test-slave-37fl"
02:07:40.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.364750 25366 fs_report.cc:389] FS layout report
02:07:40.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:07:40.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/wal
02:07:40.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/wal
02:07:40.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/data/data
02:07:40.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:07:40.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:07:40.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:07:40.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:07:40.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:07:40.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:07:40.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:07:40.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:07:40.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:07:40.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:07:40.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:07:40.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.389786 25366 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:07:40.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.390146 25366 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:07:40.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.390266 25366 kserver.cc:163] Server-wide thread pool size limit: 3276
02:07:40.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.390496 25366 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:07:40.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.391098 25387 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
02:07:40.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.392122 25366 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:07:40.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.392179 25366 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s	user 0.000s	sys 0.000s
02:07:40.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.392213 25366 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:07:40.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.392894 25366 ts_tablet_manager.cc:616] Registered 1 tablets
02:07:40.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.392939 25366 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s	user 0.000s	sys 0.000s
02:07:40.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.393061 25387 tablet_bootstrap.cc:492] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56: Bootstrap starting.
02:07:40.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.400352 25366 rpc_server.cc:307] RPC server started. Bound to: 127.22.115.3:43855
02:07:40.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.400774 25366 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskTHXZc_/test-tmp/mini-kudu-cluster11558961179868160017/ts-2/data/info.pb
02:07:40.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.404357 25387 log.cc:826] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56: Log is configured to *not* fsync() on all Append() calls
02:07:40.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.404201 25494 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.115.3:43855 every 8 connection(s)
02:07:40.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.408969 22988 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskTHXZc_/build/release/bin/kudu as pid 25366
02:07:40.409 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:07:40.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.411634 25387 tablet_bootstrap.cc:492] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56: Bootstrap replayed 1/1 log segments. Stats: ops{read=40 overwritten=0 applied=40 ignored=0} inserts{seen=30 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:07:40.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.412000 25387 tablet_bootstrap.cc:492] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56: Bootstrap complete.
02:07:40.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.412000 25497 heartbeater.cc:344] Connected to a master server at 127.22.115.61:46503
02:07:40.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.412115 25497 heartbeater.cc:461] Registering TS with master...
02:07:40.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.412317 25497 heartbeater.cc:507] Master 127.22.115.61:46503 requested a full tablet report, sending...
02:07:40.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.412796 23102 ts_manager.cc:194] Re-registered known tserver with Master: 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855)
02:07:40.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.413141 25387 ts_tablet_manager.cc:1403] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56: Time spent bootstrapping tablet: real 0.020s	user 0.008s	sys 0.008s
02:07:40.413 [DEBUG - kudu-nio-5] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563, tablet=76ab850ff70d4ccca2590ffea11986ed, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:07:40.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.413304 23102 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.115.3:60661
02:07:40.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.413995 25496 heartbeater.cc:344] Connected to a master server at 127.22.115.60:37831
02:07:40.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.414042 25496 heartbeater.cc:461] Registering TS with master...
02:07:40.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.414139 25496 heartbeater.cc:507] Master 127.22.115.60:37831 requested a full tablet report, sending...
02:07:40.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.414427 23173 ts_manager.cc:194] Re-registered known tserver with Master: 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855)
02:07:40.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.414534 25387 raft_consensus.cc:359] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 11 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:40.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.414784 25387 raft_consensus.cc:740] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [term 11 FOLLOWER]: Becoming Follower/Learner. State: Replica: 174330aa0d3e49c69cab90b20e4f7a56, State: Initialized, Role: FOLLOWER
02:07:40.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.414873 25387 consensus_queue.cc:260] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 11.40, Last appended by leader: 40, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c08b8f3e246edabc0e4e315cf9201" member_type: VOTER last_known_addr { host: "127.22.115.1" port: 46047 } } peers { permanent_uuid: "85635564f2c94c458f506c870a826b34" member_type: VOTER last_known_addr { host: "127.22.115.2" port: 43855 } } peers { permanent_uuid: "174330aa0d3e49c69cab90b20e4f7a56" member_type: VOTER last_known_addr { host: "127.22.115.3" port: 43855 } }
02:07:40.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.414956 25495 heartbeater.cc:344] Connected to a master server at 127.22.115.62:38055
02:07:40.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.415000 25495 heartbeater.cc:461] Registering TS with master...
02:07:40.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.415097 25495 heartbeater.cc:507] Master 127.22.115.62:38055 requested a full tablet report, sending...
02:07:40.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.415134 25387 ts_tablet_manager.cc:1434] T 76ab850ff70d4ccca2590ffea11986ed P 174330aa0d3e49c69cab90b20e4f7a56: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
02:07:40.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.415413 23029 ts_manager.cc:194] Re-registered known tserver with Master: 174330aa0d3e49c69cab90b20e4f7a56 (127.22.115.3:43855)
02:07:40.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.415647 25497 heartbeater.cc:499] Master 127.22.115.61:46503 was elected leader, sending a full tablet report...
02:07:40.417 [DEBUG - kudu-nio-5] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779070029563, tablet=76ab850ff70d4ccca2590ffea11986ed, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:07:40.422 [DEBUG - main] (Connection.java:575) [peer master-127.22.115.62:38055(127.22.115.62:38055)] disconnecting while in state READY
02:07:40.423 [DEBUG - main] (Connection.java:575) [peer master-127.22.115.61:46503(127.22.115.61:46503)] disconnecting while in state READY
02:07:40.424 [DEBUG - main] (Connection.java:575) [peer master-127.22.115.60:37831(127.22.115.60:37831)] disconnecting while in state READY
02:07:40.424 [DEBUG - main] (Connection.java:575) [peer 174330aa0d3e49c69cab90b20e4f7a56(127.22.115.3:43855)] disconnecting while in state TERMINATED
02:07:40.424 [DEBUG - main] (Connection.java:575) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] disconnecting while in state READY
02:07:40.425 [DEBUG - main] (Connection.java:575) [peer 619c08b8f3e246edabc0e4e315cf9201(127.22.115.1:46047)] disconnecting while in state TERMINATED
02:07:40.426 [DEBUG - kudu-nio-2] (Connection.java:239) [peer master-127.22.115.61:46503(127.22.115.61:46503)] handling channelInactive
02:07:40.426 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.22.115.62:38055(127.22.115.62:38055)] handling channelInactive
02:07:40.427 [DEBUG - kudu-nio-2] (Connection.java:709) [peer master-127.22.115.61:46503(127.22.115.61:46503)] cleaning up while in state READY due to: connection closed
02:07:40.427 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.22.115.62:38055(127.22.115.62:38055)] cleaning up while in state READY due to: connection closed
02:07:40.427 [DEBUG - kudu-nio-5] (Connection.java:239) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] handling channelInactive
02:07:40.427 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 85635564f2c94c458f506c870a826b34(127.22.115.2:43855)] cleaning up while in state READY due to: connection closed
02:07:40.427 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.22.115.60:37831(127.22.115.60:37831)] handling channelInactive
02:07:40.427 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.22.115.60:37831(127.22.115.60:37831)] cleaning up while in state READY due to: connection closed
02:07:40.428 [DEBUG - kudu-nio-4] (AsyncKuduClient.java:2704) Releasing all remaining resources
02:07:40.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.431993 22988 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskTHXZc_/build/release/bin/kudu with pid 25199
02:07:40.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.437474 22988 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskTHXZc_/build/release/bin/kudu with pid 25030
02:07:40.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.442639 22988 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskTHXZc_/build/release/bin/kudu with pid 25366
02:07:40.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.448231 22988 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskTHXZc_/build/release/bin/kudu with pid 22999
02:07:40.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.453405 22988 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskTHXZc_/build/release/bin/kudu with pid 23071
02:07:40.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:07:40.458503 22988 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskTHXZc_/build/release/bin/kudu with pid 23142
02:07:40.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-05-18T02:07:40Z chronyd exiting

Time: 33.455

OK (1 test)

