JUnit version 4.13.2
.07:59:42.894 [DEBUG - main] (RandomUtils.java:49) Using random seed: 1779091182885
07:59:42.949 [INFO - main] (RetryRule.java:138) Creating RetryStatement with result reporter and retry count of 0 (no retries)
07:59:43.013 [INFO - main] (KuduTestHarness.java:155) Creating a new MiniKuduCluster...
07:59:43.028 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskivGXg8/test-tmp
07:59:43.035 [INFO - main] (KuduBinaryLocator.java:83) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/asan/bin
07:59:43.039 [INFO - main] (MiniKuduCluster.java:217) Starting process: [/tmp/dist-test-taskivGXg8/java/build/dist-test/../../../build/asan/bin/kudu, test, mini_cluster, --serialization=pb]
07:59:43.247 [DEBUG - main] (MiniKuduCluster.java:185) Request: create_cluster {
  num_masters: 3
  num_tservers: 3
  enable_kerberos: false
  cluster_root: "/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733"
  hms_mode: NONE
  mini_kdc_options {
  }
  principal: "kudu"
  mini_oidc_options {
  }
}

07:59:43.452 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
07:59:43.454 [DEBUG - main] (MiniKuduCluster.java:185) Request: start_cluster {
}

07:59:43.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:43.457831  8310 mini_oidc.cc:150] Starting JWKS server
07:59:43.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:43.461159  8310 webserver.cc:533] Webserver started at http://127.0.0.1:34351/ using document root <none> and password file <none>
07:59:43.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:43.461488  8310 mini_oidc.cc:186] Starting OIDC Discovery server
07:59:43.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:43.462860  8310 webserver.cc:533] Webserver started at http://127.0.0.1:46351/ using document root <none> and password file <none>
07:59:43.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-05-18T07:59:43Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
07:59:43.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-05-18T07:59:43Z Disabled control of system clock
07:59:43.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:43.495604  8310 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskivGXg8/build/asan/bin/kudu
07:59:43.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskivGXg8/build/asan/bin/kudu
07:59:43.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-0/wal
07:59:43.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-0/data
07:59:43.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
07:59:43.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
07:59:43.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
07:59:43.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:59:43.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:59:43.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:59:43.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-0/logs
07:59:43.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-0/data/info.pb
07:59:43.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:59:43.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
07:59:43.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
07:59:43.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
07:59:43.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
07:59:43.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:59:43.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:59:43.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:59:43.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
07:59:43.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
07:59:43.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
07:59:43.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
07:59:43.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.8.29.190:44257
07:59:43.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.8.29.190
07:59:43.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
07:59:43.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.8.29.148:45029
07:59:43.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:59:43.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:59:43.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:59:43.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
07:59:43.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.8.29.190:44257,127.8.29.189:39965,127.8.29.188:37745
07:59:43.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
07:59:43.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34351/jwks.json with env {}
07:59:43.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:43.834955  8322 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
07:59:43.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:43.835285  8322 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
07:59:43.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:43.835361  8322 flags.cc:432] Enabled unsafe flag: --never_fsync=true
07:59:43.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:43.844817  8322 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
07:59:43.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:43.844923  8322 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
07:59:43.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:43.844978  8322 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
07:59:43.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:43.845018  8322 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
07:59:43.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:43.845072  8322 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
07:59:43.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:43.845104  8322 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:34351/jwks.json
07:59:43.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:43.856146  8322 master_runner.cc:387] Master server non-default flags:
07:59:43.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:59:43.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.8.29.148:45029
07:59:43.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:59:43.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:59:43.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-0/data
07:59:43.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-0/wal
07:59:43.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
07:59:43.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.8.29.190:44257,127.8.29.189:39965,127.8.29.188:37745
07:59:43.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:59:43.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:59:43.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
07:59:43.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.8.29.190:44257
07:59:43.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
07:59:43.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
07:59:43.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
07:59:43.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34351/jwks.json
07:59:43.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:59:43.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:59:43.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-0/data/info.pb
07:59:43.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.8.29.190
07:59:43.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
07:59:43.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
07:59:43.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:59:43.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
07:59:43.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
07:59:43.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:59:43.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-0/logs
07:59:43.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:59:43.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
07:59:43.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
07:59:43.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
07:59:43.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
07:59:43.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
07:59:43.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
07:59:43.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 07:43:24 UTC on bdcb31816ec0
07:59:43.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11954
07:59:43.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
07:59:43.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:43.858032  8322 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
07:59:43.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:43.860917  8322 file_cache.cc:492] Constructed file cache file cache with capacity 419430
07:59:43.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:43.872562  8330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:59:43.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:43.872721  8322 server_base.cc:1061] running on GCE node
07:59:43.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:43.872591  8327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:59:43.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:43.872522  8328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:59:43.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:43.874130  8322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
07:59:43.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:43.875804  8322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
07:59:43.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:43.877046  8322 hybrid_clock.cc:648] HybridClock initialized: now 1779091183876977 us; error 50 us; skew 500 ppm
07:59:43.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:43.877382  8322 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
07:59:43.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:43.879727  8322 webserver.cc:533] Webserver started at http://127.8.29.190:44149/ using document root <none> and password file <none>
07:59:43.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:43.880472  8322 fs_manager.cc:362] Metadata directory not provided
07:59:43.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:43.880585  8322 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
07:59:43.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:43.880959  8322 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
07:59:43.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:43.883574  8322 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-0/data/instance:
07:59:43.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "193be73781b34ef7b108f6c318247726"
07:59:43.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 07:59:43 on dist-test-slave-c1kb"
07:59:43.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:43.884472  8322 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-0/wal/instance:
07:59:43.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "193be73781b34ef7b108f6c318247726"
07:59:43.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 07:59:43 on dist-test-slave-c1kb"
07:59:43.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:43.890587  8322 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.006s	sys 0.000s
07:59:43.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:43.894472  8336 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
07:59:43.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:43.896272  8322 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
07:59:43.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:43.896479  8322 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-0/data,/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-0/wal
07:59:43.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "193be73781b34ef7b108f6c318247726"
07:59:43.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 07:59:43 on dist-test-slave-c1kb"
07:59:43.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:43.896718  8322 fs_report.cc:389] FS layout report
07:59:43.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
07:59:43.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-0/wal
07:59:43.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-0/wal
07:59:43.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-0/data/data
07:59:43.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
07:59:43.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
07:59:43.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
07:59:43.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
07:59:43.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
07:59:43.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
07:59:43.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
07:59:43.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
07:59:43.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
07:59:43.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
07:59:43.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
07:59:43.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:43.917351  8322 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
07:59:43.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:43.918879  8322 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
07:59:43.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:43.919132  8322 kserver.cc:163] Server-wide thread pool size limit: 3276
07:59:43.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:43.939970  8322 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.190:44257
07:59:43.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:43.939929  8387 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.29.190:44257 every 8 connection(s)
07:59:43.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:43.941776  8322 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-0/data/info.pb
07:59:43.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:43.946378  8388 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
07:59:43.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:43.947445  8310 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskivGXg8/build/asan/bin/kudu as pid 8322
07:59:43.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:43.947743  8310 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-0/wal/instance
07:59:43.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:43.949954  8310 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskivGXg8/build/asan/bin/kudu
07:59:43.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskivGXg8/build/asan/bin/kudu
07:59:43.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-1/wal
07:59:43.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-1/data
07:59:43.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
07:59:43.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
07:59:43.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
07:59:43.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:59:43.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:59:43.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:59:43.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-1/logs
07:59:43.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-1/data/info.pb
07:59:43.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:59:43.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
07:59:43.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
07:59:43.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
07:59:43.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
07:59:43.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:59:43.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:59:43.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:59:43.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
07:59:43.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
07:59:43.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
07:59:43.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
07:59:43.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.8.29.189:39965
07:59:43.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.8.29.189
07:59:43.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
07:59:43.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.8.29.148:45029
07:59:43.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:59:43.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:59:43.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:59:43.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
07:59:43.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.8.29.190:44257,127.8.29.189:39965,127.8.29.188:37745
07:59:43.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
07:59:43.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34351/jwks.json with env {}
07:59:43.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:43.954061  8388 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.29.190" port: 44257 } has no permanent_uuid. Determining permanent_uuid...
07:59:43.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:43.974411  8388 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.29.189" port: 39965 } has no permanent_uuid. Determining permanent_uuid...
07:59:43.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:43.975512  8339 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.29.189:39965: connect: Connection refused (error 111)
07:59:43.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:43.977123  8388 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.29.189:39965: Network error: Client connection negotiation failed: client connection to 127.8.29.189:39965: connect: Connection refused (error 111)
07:59:44.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.026525  8388 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.189" port: 39965 } attempt: 1
07:59:44.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:44.028645  8388 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.29.189:39965: Network error: Client connection negotiation failed: client connection to 127.8.29.189:39965: connect: Connection refused (error 111)
07:59:44.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.096939  8388 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.189" port: 39965 } attempt: 2
07:59:44.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:44.099597  8388 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.29.189:39965: Network error: Client connection negotiation failed: client connection to 127.8.29.189:39965: connect: Connection refused (error 111)
07:59:44.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.190939  8388 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.189" port: 39965 } attempt: 3
07:59:44.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:44.193151  8388 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.29.189:39965: Network error: Client connection negotiation failed: client connection to 127.8.29.189:39965: connect: Connection refused (error 111)
07:59:44.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:44.286458  8390 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
07:59:44.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:44.286782  8390 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
07:59:44.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:44.286859  8390 flags.cc:432] Enabled unsafe flag: --never_fsync=true
07:59:44.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:44.295956  8390 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
07:59:44.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:44.296061  8390 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
07:59:44.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:44.296115  8390 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
07:59:44.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:44.296155  8390 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
07:59:44.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:44.296206  8390 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
07:59:44.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:44.296239  8390 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:34351/jwks.json
07:59:44.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.306596  8390 master_runner.cc:387] Master server non-default flags:
07:59:44.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:59:44.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.8.29.148:45029
07:59:44.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:59:44.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:59:44.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-1/data
07:59:44.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-1/wal
07:59:44.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
07:59:44.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.8.29.190:44257,127.8.29.189:39965,127.8.29.188:37745
07:59:44.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:59:44.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:59:44.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
07:59:44.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.8.29.189:39965
07:59:44.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
07:59:44.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
07:59:44.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
07:59:44.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34351/jwks.json
07:59:44.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:59:44.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:59:44.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-1/data/info.pb
07:59:44.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.8.29.189
07:59:44.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
07:59:44.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
07:59:44.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:59:44.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
07:59:44.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
07:59:44.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:59:44.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-1/logs
07:59:44.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:59:44.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
07:59:44.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
07:59:44.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
07:59:44.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
07:59:44.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
07:59:44.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
07:59:44.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 07:43:24 UTC on bdcb31816ec0
07:59:44.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11954
07:59:44.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
07:59:44.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.308638  8390 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
07:59:44.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.310714  8390 file_cache.cc:492] Constructed file cache file cache with capacity 419430
07:59:44.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:44.320779  8398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:59:44.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:44.320834  8399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:59:44.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:44.322177  8401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:59:44.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.323786  8390 server_base.cc:1061] running on GCE node
07:59:44.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.324630  8390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
07:59:44.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.326588  8390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
07:59:44.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.327848  8390 hybrid_clock.cc:648] HybridClock initialized: now 1779091184327746 us; error 113 us; skew 500 ppm
07:59:44.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.328347  8390 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
07:59:44.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.330899  8390 webserver.cc:533] Webserver started at http://127.8.29.189:43825/ using document root <none> and password file <none>
07:59:44.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.331818  8390 fs_manager.cc:362] Metadata directory not provided
07:59:44.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.331960  8390 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
07:59:44.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.332429  8390 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
07:59:44.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.334949  8390 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-1/data/instance:
07:59:44.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d4454aeaafbf474d95d92774b5501d63"
07:59:44.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 07:59:44 on dist-test-slave-c1kb"
07:59:44.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.336058  8390 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-1/wal/instance:
07:59:44.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d4454aeaafbf474d95d92774b5501d63"
07:59:44.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 07:59:44 on dist-test-slave-c1kb"
07:59:44.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.336416  8388 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.189" port: 39965 } attempt: 4
07:59:44.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:44.338600  8388 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.29.189:39965: Network error: Client connection negotiation failed: client connection to 127.8.29.189:39965: connect: Connection refused (error 111)
07:59:44.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.342518  8390 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.005s	sys 0.001s
07:59:44.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.346375  8407 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
07:59:44.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.348003  8390 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.002s
07:59:44.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.348250  8390 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-1/data,/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-1/wal
07:59:44.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d4454aeaafbf474d95d92774b5501d63"
07:59:44.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 07:59:44 on dist-test-slave-c1kb"
07:59:44.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.348491  8390 fs_report.cc:389] FS layout report
07:59:44.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
07:59:44.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-1/wal
07:59:44.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-1/wal
07:59:44.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-1/data/data
07:59:44.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
07:59:44.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
07:59:44.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
07:59:44.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
07:59:44.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
07:59:44.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
07:59:44.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
07:59:44.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
07:59:44.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
07:59:44.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
07:59:44.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
07:59:44.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.377749  8390 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
07:59:44.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.378973  8390 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
07:59:44.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.379321  8390 kserver.cc:163] Server-wide thread pool size limit: 3276
07:59:44.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.401634  8390 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.189:39965
07:59:44.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.401641  8458 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.29.189:39965 every 8 connection(s)
07:59:44.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.403381  8390 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-1/data/info.pb
07:59:44.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.407861  8459 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
07:59:44.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.408617  8310 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskivGXg8/build/asan/bin/kudu as pid 8390
07:59:44.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.408795  8310 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-1/wal/instance
07:59:44.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.410027  8310 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskivGXg8/build/asan/bin/kudu
07:59:44.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskivGXg8/build/asan/bin/kudu
07:59:44.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-2/wal
07:59:44.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-2/data
07:59:44.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
07:59:44.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
07:59:44.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
07:59:44.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:59:44.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:59:44.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:59:44.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-2/logs
07:59:44.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-2/data/info.pb
07:59:44.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:59:44.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
07:59:44.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
07:59:44.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
07:59:44.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
07:59:44.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:59:44.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:59:44.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:59:44.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
07:59:44.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
07:59:44.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
07:59:44.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
07:59:44.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.8.29.188:37745
07:59:44.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.8.29.188
07:59:44.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
07:59:44.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.8.29.148:45029
07:59:44.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:59:44.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:59:44.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:59:44.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
07:59:44.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.8.29.190:44257,127.8.29.189:39965,127.8.29.188:37745
07:59:44.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
07:59:44.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34351/jwks.json with env {}
07:59:44.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.417915  8459 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.29.190" port: 44257 } has no permanent_uuid. Determining permanent_uuid...
07:59:44.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.435751  8459 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.29.189" port: 39965 } has no permanent_uuid. Determining permanent_uuid...
07:59:44.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.445305  8459 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.29.188" port: 37745 } has no permanent_uuid. Determining permanent_uuid...
07:59:44.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:44.446266  8409 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.29.188:37745: connect: Connection refused (error 111)
07:59:44.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:44.447817  8459 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.29.188:37745: Network error: Client connection negotiation failed: client connection to 127.8.29.188:37745: connect: Connection refused (error 111)
07:59:44.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.497166  8459 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.188" port: 37745 } attempt: 1
07:59:44.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:44.499214  8459 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.29.188:37745: Network error: Client connection negotiation failed: client connection to 127.8.29.188:37745: connect: Connection refused (error 111)
07:59:44.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.567513  8459 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.188" port: 37745 } attempt: 2
07:59:44.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:44.569842  8459 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.29.188:37745: Network error: Client connection negotiation failed: client connection to 127.8.29.188:37745: connect: Connection refused (error 111)
07:59:44.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.637887  8388 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.189" port: 39965 } attempt: 5
07:59:44.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.645243  8388 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.29.188" port: 37745 } has no permanent_uuid. Determining permanent_uuid...
07:59:44.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:44.647451  8388 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.29.188:37745: Network error: Client connection negotiation failed: client connection to 127.8.29.188:37745: connect: Connection refused (error 111)
07:59:44.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.661087  8459 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.188" port: 37745 } attempt: 3
07:59:44.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:44.663293  8459 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.29.188:37745: Network error: Client connection negotiation failed: client connection to 127.8.29.188:37745: connect: Connection refused (error 111)
07:59:44.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.698700  8388 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.188" port: 37745 } attempt: 1
07:59:44.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:44.700927  8388 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.29.188:37745: Network error: Client connection negotiation failed: client connection to 127.8.29.188:37745: connect: Connection refused (error 111)
07:59:44.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:44.750463  8461 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
07:59:44.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:44.750782  8461 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
07:59:44.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:44.750854  8461 flags.cc:432] Enabled unsafe flag: --never_fsync=true
07:59:44.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:44.760496  8461 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
07:59:44.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:44.760600  8461 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
07:59:44.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:44.760654  8461 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
07:59:44.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:44.760694  8461 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
07:59:44.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:44.760746  8461 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
07:59:44.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:44.760779  8461 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:34351/jwks.json
07:59:44.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.769181  8388 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.188" port: 37745 } attempt: 2
07:59:44.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:44.771299  8388 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.29.188:37745: Network error: Client connection negotiation failed: client connection to 127.8.29.188:37745: connect: Connection refused (error 111)
07:59:44.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.771745  8461 master_runner.cc:387] Master server non-default flags:
07:59:44.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:59:44.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.8.29.148:45029
07:59:44.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:59:44.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:59:44.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-2/data
07:59:44.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-2/wal
07:59:44.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
07:59:44.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.8.29.190:44257,127.8.29.189:39965,127.8.29.188:37745
07:59:44.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:59:44.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:59:44.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
07:59:44.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.8.29.188:37745
07:59:44.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
07:59:44.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
07:59:44.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
07:59:44.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34351/jwks.json
07:59:44.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:59:44.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:59:44.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-2/data/info.pb
07:59:44.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.8.29.188
07:59:44.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
07:59:44.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
07:59:44.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:59:44.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
07:59:44.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
07:59:44.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:59:44.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-2/logs
07:59:44.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:59:44.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
07:59:44.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
07:59:44.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
07:59:44.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
07:59:44.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
07:59:44.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
07:59:44.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 07:43:24 UTC on bdcb31816ec0
07:59:44.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11954
07:59:44.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
07:59:44.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.773367  8461 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
07:59:44.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.775287  8461 file_cache.cc:492] Constructed file cache file cache with capacity 419430
07:59:44.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:44.783936  8469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:59:44.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:44.784348  8472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:59:44.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:44.785061  8470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:59:44.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.786396  8461 server_base.cc:1061] running on GCE node
07:59:44.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.787317  8461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
07:59:44.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.788909  8461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
07:59:44.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.790141  8461 hybrid_clock.cc:648] HybridClock initialized: now 1779091184790071 us; error 60 us; skew 500 ppm
07:59:44.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.790604  8461 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
07:59:44.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.792886  8461 webserver.cc:533] Webserver started at http://127.8.29.188:33655/ using document root <none> and password file <none>
07:59:44.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.793689  8461 fs_manager.cc:362] Metadata directory not provided
07:59:44.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.793826  8461 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
07:59:44.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.794281  8461 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
07:59:44.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.796763  8461 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-2/data/instance:
07:59:44.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "95715c024fa240fcbba53856e4db11ba"
07:59:44.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 07:59:44 on dist-test-slave-c1kb"
07:59:44.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.797761  8461 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-2/wal/instance:
07:59:44.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "95715c024fa240fcbba53856e4db11ba"
07:59:44.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 07:59:44 on dist-test-slave-c1kb"
07:59:44.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.803300  8461 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.001s
07:59:44.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.806564  8459 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.188" port: 37745 } attempt: 4
07:59:44.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.807067  8478 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
07:59:44.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.808769  8461 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
07:59:44.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:44.808831  8459 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.29.188:37745: Network error: Client connection negotiation failed: client connection to 127.8.29.188:37745: connect: Connection refused (error 111)
07:59:44.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.808964  8461 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-2/data,/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-2/wal
07:59:44.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "95715c024fa240fcbba53856e4db11ba"
07:59:44.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 07:59:44 on dist-test-slave-c1kb"
07:59:44.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.809136  8461 fs_report.cc:389] FS layout report
07:59:44.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
07:59:44.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-2/wal
07:59:44.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-2/wal
07:59:44.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-2/data/data
07:59:44.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
07:59:44.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
07:59:44.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
07:59:44.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
07:59:44.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
07:59:44.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
07:59:44.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
07:59:44.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
07:59:44.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
07:59:44.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
07:59:44.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
07:59:44.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.821851  8461 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
07:59:44.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.822721  8461 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
07:59:44.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.822962  8461 kserver.cc:163] Server-wide thread pool size limit: 3276
07:59:44.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.846042  8461 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.188:37745
07:59:44.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.846061  8529 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.29.188:37745 every 8 connection(s)
07:59:44.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.847800  8461 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-2/data/info.pb
07:59:44.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.848799  8310 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskivGXg8/build/asan/bin/kudu as pid 8461
07:59:44.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.848999  8310 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/master-2/wal/instance
07:59:44.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.853812  8530 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
07:59:44.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.862399  8530 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.29.190" port: 44257 } has no permanent_uuid. Determining permanent_uuid...
07:59:44.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.877501  8388 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.188" port: 37745 } attempt: 3
07:59:44.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.877740  8530 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.29.189" port: 39965 } has no permanent_uuid. Determining permanent_uuid...
07:59:44.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.886870  8530 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.29.188" port: 37745 } has no permanent_uuid. Determining permanent_uuid...
07:59:44.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.898610  8388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 193be73781b34ef7b108f6c318247726: Bootstrap starting.
07:59:44.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.900846  8530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95715c024fa240fcbba53856e4db11ba: Bootstrap starting.
07:59:44.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.901706  8388 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 193be73781b34ef7b108f6c318247726: Neither blocks nor log segments found. Creating new log.
07:59:44.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.902745  8388 log.cc:826] T 00000000000000000000000000000000 P 193be73781b34ef7b108f6c318247726: Log is configured to *not* fsync() on all Append() calls
07:59:44.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.904838  8530 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95715c024fa240fcbba53856e4db11ba: Neither blocks nor log segments found. Creating new log.
07:59:44.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.905954  8388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 193be73781b34ef7b108f6c318247726: No bootstrap required, opened a new log
07:59:44.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.906437  8530 log.cc:826] T 00000000000000000000000000000000 P 95715c024fa240fcbba53856e4db11ba: Log is configured to *not* fsync() on all Append() calls
07:59:44.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.909525  8530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95715c024fa240fcbba53856e4db11ba: No bootstrap required, opened a new log
07:59:44.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.912247  8388 raft_consensus.cc:359] T 00000000000000000000000000000000 P 193be73781b34ef7b108f6c318247726 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "193be73781b34ef7b108f6c318247726" member_type: VOTER last_known_addr { host: "127.8.29.190" port: 44257 } } peers { permanent_uuid: "d4454aeaafbf474d95d92774b5501d63" member_type: VOTER last_known_addr { host: "127.8.29.189" port: 39965 } } peers { permanent_uuid: "95715c024fa240fcbba53856e4db11ba" member_type: VOTER last_known_addr { host: "127.8.29.188" port: 37745 } }
07:59:44.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.912547  8388 raft_consensus.cc:385] T 00000000000000000000000000000000 P 193be73781b34ef7b108f6c318247726 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
07:59:44.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.912660  8388 raft_consensus.cc:740] T 00000000000000000000000000000000 P 193be73781b34ef7b108f6c318247726 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 193be73781b34ef7b108f6c318247726, State: Initialized, Role: FOLLOWER
07:59:44.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.913352  8388 consensus_queue.cc:260] T 00000000000000000000000000000000 P 193be73781b34ef7b108f6c318247726 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "193be73781b34ef7b108f6c318247726" member_type: VOTER last_known_addr { host: "127.8.29.190" port: 44257 } } peers { permanent_uuid: "d4454aeaafbf474d95d92774b5501d63" member_type: VOTER last_known_addr { host: "127.8.29.189" port: 39965 } } peers { permanent_uuid: "95715c024fa240fcbba53856e4db11ba" member_type: VOTER last_known_addr { host: "127.8.29.188" port: 37745 } }
07:59:44.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.914323  8539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 193be73781b34ef7b108f6c318247726 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "193be73781b34ef7b108f6c318247726" member_type: VOTER last_known_addr { host: "127.8.29.190" port: 44257 } } peers { permanent_uuid: "d4454aeaafbf474d95d92774b5501d63" member_type: VOTER last_known_addr { host: "127.8.29.189" port: 39965 } } peers { permanent_uuid: "95715c024fa240fcbba53856e4db11ba" member_type: VOTER last_known_addr { host: "127.8.29.188" port: 37745 } } }
07:59:44.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.914670  8539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 193be73781b34ef7b108f6c318247726 [sys.catalog]: This master's current role is: FOLLOWER
07:59:44.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.915656  8388 sys_catalog.cc:565] T 00000000000000000000000000000000 P 193be73781b34ef7b108f6c318247726 [sys.catalog]: configured and running, proceeding with master startup.
07:59:44.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.916118  8530 raft_consensus.cc:359] T 00000000000000000000000000000000 P 95715c024fa240fcbba53856e4db11ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "193be73781b34ef7b108f6c318247726" member_type: VOTER last_known_addr { host: "127.8.29.190" port: 44257 } } peers { permanent_uuid: "d4454aeaafbf474d95d92774b5501d63" member_type: VOTER last_known_addr { host: "127.8.29.189" port: 39965 } } peers { permanent_uuid: "95715c024fa240fcbba53856e4db11ba" member_type: VOTER last_known_addr { host: "127.8.29.188" port: 37745 } }
07:59:44.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.916378  8530 raft_consensus.cc:385] T 00000000000000000000000000000000 P 95715c024fa240fcbba53856e4db11ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
07:59:44.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.916455  8530 raft_consensus.cc:740] T 00000000000000000000000000000000 P 95715c024fa240fcbba53856e4db11ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95715c024fa240fcbba53856e4db11ba, State: Initialized, Role: FOLLOWER
07:59:44.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.917018  8530 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95715c024fa240fcbba53856e4db11ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "193be73781b34ef7b108f6c318247726" member_type: VOTER last_known_addr { host: "127.8.29.190" port: 44257 } } peers { permanent_uuid: "d4454aeaafbf474d95d92774b5501d63" member_type: VOTER last_known_addr { host: "127.8.29.189" port: 39965 } } peers { permanent_uuid: "95715c024fa240fcbba53856e4db11ba" member_type: VOTER last_known_addr { host: "127.8.29.188" port: 37745 } }
07:59:44.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.918047  8542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95715c024fa240fcbba53856e4db11ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "193be73781b34ef7b108f6c318247726" member_type: VOTER last_known_addr { host: "127.8.29.190" port: 44257 } } peers { permanent_uuid: "d4454aeaafbf474d95d92774b5501d63" member_type: VOTER last_known_addr { host: "127.8.29.189" port: 39965 } } peers { permanent_uuid: "95715c024fa240fcbba53856e4db11ba" member_type: VOTER last_known_addr { host: "127.8.29.188" port: 37745 } } }
07:59:44.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.918601  8542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95715c024fa240fcbba53856e4db11ba [sys.catalog]: This master's current role is: FOLLOWER
07:59:44.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.919291  8530 sys_catalog.cc:565] T 00000000000000000000000000000000 P 95715c024fa240fcbba53856e4db11ba [sys.catalog]: configured and running, proceeding with master startup.
07:59:44.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:44.922127  8342 tablet.cc:2404] T 00000000000000000000000000000000 P 193be73781b34ef7b108f6c318247726: Can't schedule compaction. Clean time has not been advanced past its initial value.
07:59:44.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:44.932569  8556 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 193be73781b34ef7b108f6c318247726: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
07:59:44.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:44.933462  8556 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
07:59:44.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.929896  8322 master_runner.cc:187] Error getting master registration for 127.8.29.189:39965: OK, instance_id { permanent_uuid: "d4454aeaafbf474d95d92774b5501d63" instance_seqno: 1779091184396242 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
07:59:44.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.935977  8322 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
07:59:44.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.943584  8461 master_runner.cc:187] Error getting master registration for 127.8.29.189:39965: OK, instance_id { permanent_uuid: "d4454aeaafbf474d95d92774b5501d63" instance_seqno: 1779091184396242 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
07:59:44.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:44.954025  8461 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
07:59:44.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:44.958266  8563 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 95715c024fa240fcbba53856e4db11ba: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
07:59:44.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:44.960311  8563 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
07:59:45.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.013274  8542 raft_consensus.cc:493] T 00000000000000000000000000000000 P 95715c024fa240fcbba53856e4db11ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
07:59:45.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.013590  8542 raft_consensus.cc:515] T 00000000000000000000000000000000 P 95715c024fa240fcbba53856e4db11ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "193be73781b34ef7b108f6c318247726" member_type: VOTER last_known_addr { host: "127.8.29.190" port: 44257 } } peers { permanent_uuid: "d4454aeaafbf474d95d92774b5501d63" member_type: VOTER last_known_addr { host: "127.8.29.189" port: 39965 } } peers { permanent_uuid: "95715c024fa240fcbba53856e4db11ba" member_type: VOTER last_known_addr { host: "127.8.29.188" port: 37745 } }
07:59:45.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.015494  8542 leader_election.cc:290] T 00000000000000000000000000000000 P 95715c024fa240fcbba53856e4db11ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 193be73781b34ef7b108f6c318247726 (127.8.29.190:44257), d4454aeaafbf474d95d92774b5501d63 (127.8.29.189:39965)
07:59:45.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.016136  8363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "95715c024fa240fcbba53856e4db11ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "193be73781b34ef7b108f6c318247726" is_pre_election: true
07:59:45.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.016674  8363 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 193be73781b34ef7b108f6c318247726 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95715c024fa240fcbba53856e4db11ba in term 0.
07:59:45.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.017345  8481 leader_election.cc:304] T 00000000000000000000000000000000 P 95715c024fa240fcbba53856e4db11ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 193be73781b34ef7b108f6c318247726, 95715c024fa240fcbba53856e4db11ba; no voters: 
07:59:45.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.018455  8542 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 95715c024fa240fcbba53856e4db11ba [term 0 FOLLOWER]: Leader pre-election won for term 1
07:59:45.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.018602  8542 raft_consensus.cc:493] T 00000000000000000000000000000000 P 95715c024fa240fcbba53856e4db11ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
07:59:45.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.018698  8542 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 95715c024fa240fcbba53856e4db11ba [term 0 FOLLOWER]: Advancing to term 1
07:59:45.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.022185  8542 raft_consensus.cc:515] T 00000000000000000000000000000000 P 95715c024fa240fcbba53856e4db11ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "193be73781b34ef7b108f6c318247726" member_type: VOTER last_known_addr { host: "127.8.29.190" port: 44257 } } peers { permanent_uuid: "d4454aeaafbf474d95d92774b5501d63" member_type: VOTER last_known_addr { host: "127.8.29.189" port: 39965 } } peers { permanent_uuid: "95715c024fa240fcbba53856e4db11ba" member_type: VOTER last_known_addr { host: "127.8.29.188" port: 37745 } }
07:59:45.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.023723  8363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "95715c024fa240fcbba53856e4db11ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "193be73781b34ef7b108f6c318247726"
07:59:45.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.023823  8542 leader_election.cc:290] T 00000000000000000000000000000000 P 95715c024fa240fcbba53856e4db11ba [CANDIDATE]: Term 1 election: Requested vote from peers 193be73781b34ef7b108f6c318247726 (127.8.29.190:44257), d4454aeaafbf474d95d92774b5501d63 (127.8.29.189:39965)
07:59:45.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.023962  8363 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 193be73781b34ef7b108f6c318247726 [term 0 FOLLOWER]: Advancing to term 1
07:59:45.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.027281  8363 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 193be73781b34ef7b108f6c318247726 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95715c024fa240fcbba53856e4db11ba in term 1.
07:59:45.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.027848  8481 leader_election.cc:304] T 00000000000000000000000000000000 P 95715c024fa240fcbba53856e4db11ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 193be73781b34ef7b108f6c318247726, 95715c024fa240fcbba53856e4db11ba; no voters: 
07:59:45.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.029268  8542 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 95715c024fa240fcbba53856e4db11ba [term 1 FOLLOWER]: Leader election won for term 1
07:59:45.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.039578  8542 raft_consensus.cc:697] T 00000000000000000000000000000000 P 95715c024fa240fcbba53856e4db11ba [term 1 LEADER]: Becoming Leader. State: Replica: 95715c024fa240fcbba53856e4db11ba, State: Running, Role: LEADER
07:59:45.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.041250  8542 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95715c024fa240fcbba53856e4db11ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "193be73781b34ef7b108f6c318247726" member_type: VOTER last_known_addr { host: "127.8.29.190" port: 44257 } } peers { permanent_uuid: "d4454aeaafbf474d95d92774b5501d63" member_type: VOTER last_known_addr { host: "127.8.29.189" port: 39965 } } peers { permanent_uuid: "95715c024fa240fcbba53856e4db11ba" member_type: VOTER last_known_addr { host: "127.8.29.188" port: 37745 } }
07:59:45.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.046545  8565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95715c024fa240fcbba53856e4db11ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95715c024fa240fcbba53856e4db11ba. Latest consensus state: current_term: 1 leader_uuid: "95715c024fa240fcbba53856e4db11ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "193be73781b34ef7b108f6c318247726" member_type: VOTER last_known_addr { host: "127.8.29.190" port: 44257 } } peers { permanent_uuid: "d4454aeaafbf474d95d92774b5501d63" member_type: VOTER last_known_addr { host: "127.8.29.189" port: 39965 } } peers { permanent_uuid: "95715c024fa240fcbba53856e4db11ba" member_type: VOTER last_known_addr { host: "127.8.29.188" port: 37745 } } }
07:59:45.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.047685  8565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95715c024fa240fcbba53856e4db11ba [sys.catalog]: This master's current role is: LEADER
07:59:45.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.055966  8567 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
07:59:45.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.058251  8567 catalog_manager.cc:1519] Initializing Kudu cluster ID...
07:59:45.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:45.075263  8484 tablet.cc:2404] T 00000000000000000000000000000000 P 95715c024fa240fcbba53856e4db11ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
07:59:45.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.082175  8363 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 193be73781b34ef7b108f6c318247726 [term 1 FOLLOWER]: Refusing update from remote peer 95715c024fa240fcbba53856e4db11ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
07:59:45.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.083896  8565 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 95715c024fa240fcbba53856e4db11ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "193be73781b34ef7b108f6c318247726" member_type: VOTER last_known_addr { host: "127.8.29.190" port: 44257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
07:59:45.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.108057  8459 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.188" port: 37745 } attempt: 5
07:59:45.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.110458  8539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 193be73781b34ef7b108f6c318247726 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95715c024fa240fcbba53856e4db11ba. Latest consensus state: current_term: 1 leader_uuid: "95715c024fa240fcbba53856e4db11ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "193be73781b34ef7b108f6c318247726" member_type: VOTER last_known_addr { host: "127.8.29.190" port: 44257 } } peers { permanent_uuid: "d4454aeaafbf474d95d92774b5501d63" member_type: VOTER last_known_addr { host: "127.8.29.189" port: 39965 } } peers { permanent_uuid: "95715c024fa240fcbba53856e4db11ba" member_type: VOTER last_known_addr { host: "127.8.29.188" port: 37745 } } }
07:59:45.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.110729  8539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 193be73781b34ef7b108f6c318247726 [sys.catalog]: This master's current role is: FOLLOWER
07:59:45.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.126219  8459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4454aeaafbf474d95d92774b5501d63: Bootstrap starting.
07:59:45.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.128141  8542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95715c024fa240fcbba53856e4db11ba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "95715c024fa240fcbba53856e4db11ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "193be73781b34ef7b108f6c318247726" member_type: VOTER last_known_addr { host: "127.8.29.190" port: 44257 } } peers { permanent_uuid: "d4454aeaafbf474d95d92774b5501d63" member_type: VOTER last_known_addr { host: "127.8.29.189" port: 39965 } } peers { permanent_uuid: "95715c024fa240fcbba53856e4db11ba" member_type: VOTER last_known_addr { host: "127.8.29.188" port: 37745 } } }
07:59:45.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.128450  8542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95715c024fa240fcbba53856e4db11ba [sys.catalog]: This master's current role is: LEADER
07:59:45.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.130537  8567 catalog_manager.cc:1382] Generated new cluster ID: 15912d9b905041b295d8ff3eb1789f37
07:59:45.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.130652  8567 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
07:59:45.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.133881  8568 mvcc.cc:204] Tried to move back new op lower bound from 7287157494081187840 to 7287157493940842496. Current Snapshot: MvccSnapshot[applied={T|T < 7287157494081187840 or (T in {7287157494081187840})}]
07:59:45.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.141404  8459 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4454aeaafbf474d95d92774b5501d63: Neither blocks nor log segments found. Creating new log.
07:59:45.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.142802  8459 log.cc:826] T 00000000000000000000000000000000 P d4454aeaafbf474d95d92774b5501d63: Log is configured to *not* fsync() on all Append() calls
07:59:45.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.149828  8539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 193be73781b34ef7b108f6c318247726 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "95715c024fa240fcbba53856e4db11ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "193be73781b34ef7b108f6c318247726" member_type: VOTER last_known_addr { host: "127.8.29.190" port: 44257 } } peers { permanent_uuid: "d4454aeaafbf474d95d92774b5501d63" member_type: VOTER last_known_addr { host: "127.8.29.189" port: 39965 } } peers { permanent_uuid: "95715c024fa240fcbba53856e4db11ba" member_type: VOTER last_known_addr { host: "127.8.29.188" port: 37745 } } }
07:59:45.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.150070  8539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 193be73781b34ef7b108f6c318247726 [sys.catalog]: This master's current role is: FOLLOWER
07:59:45.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.156930  8459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4454aeaafbf474d95d92774b5501d63: No bootstrap required, opened a new log
07:59:45.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.164857  8459 raft_consensus.cc:359] T 00000000000000000000000000000000 P d4454aeaafbf474d95d92774b5501d63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "193be73781b34ef7b108f6c318247726" member_type: VOTER last_known_addr { host: "127.8.29.190" port: 44257 } } peers { permanent_uuid: "d4454aeaafbf474d95d92774b5501d63" member_type: VOTER last_known_addr { host: "127.8.29.189" port: 39965 } } peers { permanent_uuid: "95715c024fa240fcbba53856e4db11ba" member_type: VOTER last_known_addr { host: "127.8.29.188" port: 37745 } }
07:59:45.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.165712  8459 raft_consensus.cc:385] T 00000000000000000000000000000000 P d4454aeaafbf474d95d92774b5501d63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
07:59:45.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.166065  8459 raft_consensus.cc:740] T 00000000000000000000000000000000 P d4454aeaafbf474d95d92774b5501d63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4454aeaafbf474d95d92774b5501d63, State: Initialized, Role: FOLLOWER
07:59:45.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.167183  8459 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4454aeaafbf474d95d92774b5501d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "193be73781b34ef7b108f6c318247726" member_type: VOTER last_known_addr { host: "127.8.29.190" port: 44257 } } peers { permanent_uuid: "d4454aeaafbf474d95d92774b5501d63" member_type: VOTER last_known_addr { host: "127.8.29.189" port: 39965 } } peers { permanent_uuid: "95715c024fa240fcbba53856e4db11ba" member_type: VOTER last_known_addr { host: "127.8.29.188" port: 37745 } }
07:59:45.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.184130  8574 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4454aeaafbf474d95d92774b5501d63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "193be73781b34ef7b108f6c318247726" member_type: VOTER last_known_addr { host: "127.8.29.190" port: 44257 } } peers { permanent_uuid: "d4454aeaafbf474d95d92774b5501d63" member_type: VOTER last_known_addr { host: "127.8.29.189" port: 39965 } } peers { permanent_uuid: "95715c024fa240fcbba53856e4db11ba" member_type: VOTER last_known_addr { host: "127.8.29.188" port: 37745 } } }
07:59:45.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.184958  8574 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4454aeaafbf474d95d92774b5501d63 [sys.catalog]: This master's current role is: FOLLOWER
07:59:45.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.185691  8459 sys_catalog.cc:565] T 00000000000000000000000000000000 P d4454aeaafbf474d95d92774b5501d63 [sys.catalog]: configured and running, proceeding with master startup.
07:59:45.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.186084  8433 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "95715c024fa240fcbba53856e4db11ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4454aeaafbf474d95d92774b5501d63"
07:59:45.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.186596  8433 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d4454aeaafbf474d95d92774b5501d63 [term 0 FOLLOWER]: Advancing to term 1
07:59:45.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.189021  8434 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "95715c024fa240fcbba53856e4db11ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4454aeaafbf474d95d92774b5501d63" is_pre_election: true
07:59:45.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.191118  8433 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d4454aeaafbf474d95d92774b5501d63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95715c024fa240fcbba53856e4db11ba in term 1.
07:59:45.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.191824  8432 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d4454aeaafbf474d95d92774b5501d63 [term 1 FOLLOWER]: Refusing update from remote peer 95715c024fa240fcbba53856e4db11ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
07:59:45.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.194520  8565 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 95715c024fa240fcbba53856e4db11ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4454aeaafbf474d95d92774b5501d63" member_type: VOTER last_known_addr { host: "127.8.29.189" port: 39965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
07:59:45.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.205766  8567 catalog_manager.cc:1405] Generated new certificate authority record
07:59:45.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.207645  8567 catalog_manager.cc:1539] Loading token signing keys...
07:59:45.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.243242  8567 catalog_manager.cc:6069] T 00000000000000000000000000000000 P 95715c024fa240fcbba53856e4db11ba: Generated new TSK 0
07:59:45.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.244778  8567 catalog_manager.cc:1549] Initializing in-progress tserver states...
07:59:45.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.259630  8565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95715c024fa240fcbba53856e4db11ba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "95715c024fa240fcbba53856e4db11ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "193be73781b34ef7b108f6c318247726" member_type: VOTER last_known_addr { host: "127.8.29.190" port: 44257 } } peers { permanent_uuid: "d4454aeaafbf474d95d92774b5501d63" member_type: VOTER last_known_addr { host: "127.8.29.189" port: 39965 } } peers { permanent_uuid: "95715c024fa240fcbba53856e4db11ba" member_type: VOTER last_known_addr { host: "127.8.29.188" port: 37745 } } }
07:59:45.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.259974  8565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95715c024fa240fcbba53856e4db11ba [sys.catalog]: This master's current role is: LEADER
07:59:45.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.260731  8310 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskivGXg8/build/asan/bin/kudu
07:59:45.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskivGXg8/build/asan/bin/kudu
07:59:45.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/wal
07:59:45.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/data
07:59:45.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
07:59:45.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
07:59:45.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
07:59:45.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:59:45.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:59:45.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:59:45.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/logs
07:59:45.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/data/info.pb
07:59:45.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:59:45.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
07:59:45.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
07:59:45.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
07:59:45.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
07:59:45.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:59:45.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:59:45.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:59:45.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
07:59:45.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
07:59:45.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.8.29.129:0
07:59:45.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.8.29.129
07:59:45.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.8.29.129
07:59:45.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
07:59:45.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.8.29.190:44257,127.8.29.189:39965,127.8.29.188:37745
07:59:45.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.8.29.148:45029
07:59:45.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:59:45.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:59:45.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
07:59:45.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.288971  8580 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4454aeaafbf474d95d92774b5501d63 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "95715c024fa240fcbba53856e4db11ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "193be73781b34ef7b108f6c318247726" member_type: VOTER last_known_addr { host: "127.8.29.190" port: 44257 } } peers { permanent_uuid: "d4454aeaafbf474d95d92774b5501d63" member_type: VOTER last_known_addr { host: "127.8.29.189" port: 39965 } } peers { permanent_uuid: "95715c024fa240fcbba53856e4db11ba" member_type: VOTER last_known_addr { host: "127.8.29.188" port: 37745 } } }
07:59:45.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.289326  8580 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4454aeaafbf474d95d92774b5501d63 [sys.catalog]: This master's current role is: FOLLOWER
07:59:45.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.290751  8574 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4454aeaafbf474d95d92774b5501d63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95715c024fa240fcbba53856e4db11ba. Latest consensus state: current_term: 1 leader_uuid: "95715c024fa240fcbba53856e4db11ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "193be73781b34ef7b108f6c318247726" member_type: VOTER last_known_addr { host: "127.8.29.190" port: 44257 } } peers { permanent_uuid: "d4454aeaafbf474d95d92774b5501d63" member_type: VOTER last_known_addr { host: "127.8.29.189" port: 39965 } } peers { permanent_uuid: "95715c024fa240fcbba53856e4db11ba" member_type: VOTER last_known_addr { host: "127.8.29.188" port: 37745 } } }
07:59:45.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.290985  8574 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4454aeaafbf474d95d92774b5501d63 [sys.catalog]: This master's current role is: FOLLOWER
07:59:45.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.410893  8591 catalog_manager.cc:1294] Loaded cluster ID: 15912d9b905041b295d8ff3eb1789f37
07:59:45.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.411620  8591 catalog_manager.cc:1587] T 00000000000000000000000000000000 P d4454aeaafbf474d95d92774b5501d63: loading cluster ID for follower catalog manager: success
07:59:45.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.416450  8591 catalog_manager.cc:1609] T 00000000000000000000000000000000 P d4454aeaafbf474d95d92774b5501d63: acquiring CA information for follower catalog manager: success
07:59:45.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.433295  8591 catalog_manager.cc:1637] T 00000000000000000000000000000000 P d4454aeaafbf474d95d92774b5501d63: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
07:59:45.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:45.776153  8582 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
07:59:45.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:45.776516  8582 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
07:59:45.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:45.776618  8582 flags.cc:432] Enabled unsafe flag: --never_fsync=true
07:59:45.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:45.786088  8582 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
07:59:45.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:45.786336  8582 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.129
07:59:45.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.797254  8582 tablet_server_runner.cc:78] Tablet server non-default flags:
07:59:45.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:59:45.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.8.29.148:45029
07:59:45.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:59:45.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:59:45.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/data
07:59:45.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/wal
07:59:45.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:59:45.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:59:45.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.8.29.129:0
07:59:45.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
07:59:45.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:59:45.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:59:45.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/data/info.pb
07:59:45.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.8.29.129
07:59:45.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
07:59:45.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.8.29.190:44257,127.8.29.189:39965,127.8.29.188:37745
07:59:45.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
07:59:45.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:59:45.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
07:59:45.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
07:59:45.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:59:45.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.8.29.129
07:59:45.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/logs
07:59:45.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:59:45.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
07:59:45.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
07:59:45.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
07:59:45.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
07:59:45.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
07:59:45.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
07:59:45.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 07:43:24 UTC on bdcb31816ec0
07:59:45.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11954
07:59:45.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
07:59:45.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.801057  8582 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
07:59:45.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.804494  8582 file_cache.cc:492] Constructed file cache file cache with capacity 419430
07:59:45.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:45.816555  8597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:59:45.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:45.817036  8598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:59:45.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:45.829401  8600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:59:45.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.829634  8582 server_base.cc:1061] running on GCE node
07:59:45.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.830907  8582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
07:59:45.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.832444  8582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
07:59:45.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.833916  8582 hybrid_clock.cc:648] HybridClock initialized: now 1779091185833814 us; error 94 us; skew 500 ppm
07:59:45.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.834298  8582 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
07:59:45.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.837144  8582 webserver.cc:533] Webserver started at http://127.8.29.129:38159/ using document root <none> and password file <none>
07:59:45.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.838150  8582 fs_manager.cc:362] Metadata directory not provided
07:59:45.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.838275  8582 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
07:59:45.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.838660  8582 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
07:59:45.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.841848  8582 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/data/instance:
07:59:45.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "bcfe780000f44bb5ba6ab6347414c393"
07:59:45.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 07:59:45 on dist-test-slave-c1kb"
07:59:45.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.842813  8582 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/wal/instance:
07:59:45.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "bcfe780000f44bb5ba6ab6347414c393"
07:59:45.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 07:59:45 on dist-test-slave-c1kb"
07:59:45.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.850062  8582 fs_manager.cc:696] Time spent creating directory manager: real 0.007s	user 0.009s	sys 0.000s
07:59:45.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.854606  8606 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
07:59:45.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.856892  8582 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.001s	sys 0.002s
07:59:45.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.857080  8582 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/data,/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/wal
07:59:45.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "bcfe780000f44bb5ba6ab6347414c393"
07:59:45.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 07:59:45 on dist-test-slave-c1kb"
07:59:45.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.857339  8582 fs_report.cc:389] FS layout report
07:59:45.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
07:59:45.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/wal
07:59:45.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/wal
07:59:45.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/data/data
07:59:45.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
07:59:45.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
07:59:45.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
07:59:45.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
07:59:45.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
07:59:45.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
07:59:45.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
07:59:45.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
07:59:45.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
07:59:45.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
07:59:45.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
07:59:45.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.876420  8582 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
07:59:45.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.877452  8582 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
07:59:45.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.877729  8582 kserver.cc:163] Server-wide thread pool size limit: 3276
07:59:45.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.878866  8582 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
07:59:45.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.881076  8582 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
07:59:45.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.881213  8582 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
07:59:45.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.881315  8582 ts_tablet_manager.cc:616] Registered 0 tablets
07:59:45.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.881382  8582 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
07:59:45.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.917487  8582 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.129:44331
07:59:45.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.917590  8718 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.29.129:44331 every 8 connection(s)
07:59:45.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.919301  8582 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/data/info.pb
07:59:45.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.928442  8310 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskivGXg8/build/asan/bin/kudu as pid 8582
07:59:45.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.928690  8310 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/wal/instance
07:59:45.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.934494  8310 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskivGXg8/build/asan/bin/kudu
07:59:45.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskivGXg8/build/asan/bin/kudu
07:59:45.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/wal
07:59:45.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/data
07:59:45.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
07:59:45.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
07:59:45.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
07:59:45.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:59:45.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:59:45.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:59:45.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/logs
07:59:45.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/data/info.pb
07:59:45.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:59:45.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
07:59:45.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
07:59:45.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
07:59:45.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
07:59:45.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:59:45.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:59:45.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:59:45.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
07:59:45.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
07:59:45.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.8.29.130:0
07:59:45.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.8.29.130
07:59:45.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.8.29.130
07:59:45.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
07:59:45.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.8.29.190:44257,127.8.29.189:39965,127.8.29.188:37745
07:59:45.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.8.29.148:45029
07:59:45.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:59:45.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:59:45.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
07:59:45.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.936187  8556 catalog_manager.cc:1294] Loaded cluster ID: 15912d9b905041b295d8ff3eb1789f37
07:59:45.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.936288  8556 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 193be73781b34ef7b108f6c318247726: loading cluster ID for follower catalog manager: success
07:59:45.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.940475  8556 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 193be73781b34ef7b108f6c318247726: acquiring CA information for follower catalog manager: success
07:59:45.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.950253  8556 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 193be73781b34ef7b108f6c318247726: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
07:59:45.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.953301  8719 heartbeater.cc:344] Connected to a master server at 127.8.29.188:37745
07:59:45.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.953671  8719 heartbeater.cc:461] Registering TS with master...
07:59:45.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.954732  8719 heartbeater.cc:507] Master 127.8.29.188:37745 requested a full tablet report, sending...
07:59:45.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.957130  8495 ts_manager.cc:194] Registered new tserver with Master: bcfe780000f44bb5ba6ab6347414c393 (127.8.29.129:44331)
07:59:45.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.959226  8495 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.8.29.129:45695
07:59:45.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.963826  8720 heartbeater.cc:344] Connected to a master server at 127.8.29.190:44257
07:59:45.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.963990  8720 heartbeater.cc:461] Registering TS with master...
07:59:45.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.964370  8720 heartbeater.cc:507] Master 127.8.29.190:44257 requested a full tablet report, sending...
07:59:45.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.966151  8353 ts_manager.cc:194] Registered new tserver with Master: bcfe780000f44bb5ba6ab6347414c393 (127.8.29.129:44331)
07:59:45.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.969045  8721 heartbeater.cc:344] Connected to a master server at 127.8.29.189:39965
07:59:45.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.969197  8721 heartbeater.cc:461] Registering TS with master...
07:59:45.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.969501  8721 heartbeater.cc:507] Master 127.8.29.189:39965 requested a full tablet report, sending...
07:59:45.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:45.971710  8424 ts_manager.cc:194] Registered new tserver with Master: bcfe780000f44bb5ba6ab6347414c393 (127.8.29.129:44331)
07:59:46.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:46.285275  8729 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
07:59:46.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:46.285596  8729 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
07:59:46.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:46.285666  8729 flags.cc:432] Enabled unsafe flag: --never_fsync=true
07:59:46.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:46.294466  8729 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
07:59:46.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:46.294656  8729 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.130
07:59:46.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.305243  8729 tablet_server_runner.cc:78] Tablet server non-default flags:
07:59:46.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:59:46.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.8.29.148:45029
07:59:46.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:59:46.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:59:46.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/data
07:59:46.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/wal
07:59:46.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:59:46.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:59:46.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.8.29.130:0
07:59:46.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
07:59:46.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:59:46.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:59:46.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/data/info.pb
07:59:46.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.8.29.130
07:59:46.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
07:59:46.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.8.29.190:44257,127.8.29.189:39965,127.8.29.188:37745
07:59:46.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
07:59:46.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:59:46.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
07:59:46.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
07:59:46.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:59:46.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.8.29.130
07:59:46.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/logs
07:59:46.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:59:46.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
07:59:46.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
07:59:46.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
07:59:46.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
07:59:46.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
07:59:46.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
07:59:46.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 07:43:24 UTC on bdcb31816ec0
07:59:46.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11954
07:59:46.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
07:59:46.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.306908  8729 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
07:59:46.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.309345  8729 file_cache.cc:492] Constructed file cache file cache with capacity 419430
07:59:46.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:46.321317  8734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:59:46.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:46.321395  8737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:59:46.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:46.321807  8735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:59:46.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.323531  8729 server_base.cc:1061] running on GCE node
07:59:46.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.324152  8729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
07:59:46.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.326505  8729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
07:59:46.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.327742  8729 hybrid_clock.cc:648] HybridClock initialized: now 1779091186327674 us; error 57 us; skew 500 ppm
07:59:46.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.328176  8729 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
07:59:46.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.334676  8729 webserver.cc:533] Webserver started at http://127.8.29.130:44565/ using document root <none> and password file <none>
07:59:46.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.335752  8729 fs_manager.cc:362] Metadata directory not provided
07:59:46.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.335863  8729 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
07:59:46.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.336198  8729 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
07:59:46.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.338650  8729 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/data/instance:
07:59:46.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "21124dcbd9914973a616d9ed69870d0c"
07:59:46.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 07:59:46 on dist-test-slave-c1kb"
07:59:46.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.339494  8729 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/wal/instance:
07:59:46.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "21124dcbd9914973a616d9ed69870d0c"
07:59:46.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 07:59:46 on dist-test-slave-c1kb"
07:59:46.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.345404  8729 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
07:59:46.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.349233  8743 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
07:59:46.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.350805  8729 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
07:59:46.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.350952  8729 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/data,/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/wal
07:59:46.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "21124dcbd9914973a616d9ed69870d0c"
07:59:46.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 07:59:46 on dist-test-slave-c1kb"
07:59:46.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.351116  8729 fs_report.cc:389] FS layout report
07:59:46.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
07:59:46.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/wal
07:59:46.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/wal
07:59:46.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/data/data
07:59:46.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
07:59:46.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
07:59:46.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
07:59:46.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
07:59:46.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
07:59:46.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
07:59:46.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
07:59:46.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
07:59:46.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
07:59:46.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
07:59:46.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
07:59:46.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.378051  8729 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
07:59:46.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.378948  8729 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
07:59:46.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.379202  8729 kserver.cc:163] Server-wide thread pool size limit: 3276
07:59:46.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.380224  8729 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
07:59:46.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.382081  8729 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
07:59:46.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.382181  8729 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
07:59:46.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.382259  8729 ts_tablet_manager.cc:616] Registered 0 tablets
07:59:46.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.382313  8729 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
07:59:46.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.418699  8729 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.130:37019
07:59:46.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.418784  8855 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.29.130:37019 every 8 connection(s)
07:59:46.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.420608  8729 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/data/info.pb
07:59:46.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.429736  8310 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskivGXg8/build/asan/bin/kudu as pid 8729
07:59:46.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.429951  8310 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/wal/instance
07:59:46.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.435552  8310 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskivGXg8/build/asan/bin/kudu
07:59:46.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskivGXg8/build/asan/bin/kudu
07:59:46.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/wal
07:59:46.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/data
07:59:46.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
07:59:46.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
07:59:46.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
07:59:46.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:59:46.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:59:46.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:59:46.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/logs
07:59:46.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/data/info.pb
07:59:46.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:59:46.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
07:59:46.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
07:59:46.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
07:59:46.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
07:59:46.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:59:46.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:59:46.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:59:46.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
07:59:46.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
07:59:46.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.8.29.131:0
07:59:46.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.8.29.131
07:59:46.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.8.29.131
07:59:46.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
07:59:46.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.8.29.190:44257,127.8.29.189:39965,127.8.29.188:37745
07:59:46.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.8.29.148:45029
07:59:46.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:59:46.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:59:46.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
07:59:46.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.441171  8856 heartbeater.cc:344] Connected to a master server at 127.8.29.188:37745
07:59:46.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.441682  8856 heartbeater.cc:461] Registering TS with master...
07:59:46.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.443042  8856 heartbeater.cc:507] Master 127.8.29.188:37745 requested a full tablet report, sending...
07:59:46.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.445214  8495 ts_manager.cc:194] Registered new tserver with Master: 21124dcbd9914973a616d9ed69870d0c (127.8.29.130:37019)
07:59:46.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.446420  8495 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.8.29.130:47893
07:59:46.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.446976  8858 heartbeater.cc:344] Connected to a master server at 127.8.29.189:39965
07:59:46.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.447003  8857 heartbeater.cc:344] Connected to a master server at 127.8.29.190:44257
07:59:46.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.447172  8857 heartbeater.cc:461] Registering TS with master...
07:59:46.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.447176  8858 heartbeater.cc:461] Registering TS with master...
07:59:46.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.447842  8857 heartbeater.cc:507] Master 127.8.29.190:44257 requested a full tablet report, sending...
07:59:46.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.447817  8858 heartbeater.cc:507] Master 127.8.29.189:39965 requested a full tablet report, sending...
07:59:46.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.449031  8424 ts_manager.cc:194] Registered new tserver with Master: 21124dcbd9914973a616d9ed69870d0c (127.8.29.130:37019)
07:59:46.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.449754  8353 ts_manager.cc:194] Registered new tserver with Master: 21124dcbd9914973a616d9ed69870d0c (127.8.29.130:37019)
07:59:46.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:46.781649  8863 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
07:59:46.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:46.782042  8863 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
07:59:46.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:46.782168  8863 flags.cc:432] Enabled unsafe flag: --never_fsync=true
07:59:46.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:46.791697  8863 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
07:59:46.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:46.791925  8863 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.131
07:59:46.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.802049  8863 tablet_server_runner.cc:78] Tablet server non-default flags:
07:59:46.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:59:46.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.8.29.148:45029
07:59:46.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:59:46.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:59:46.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/data
07:59:46.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/wal
07:59:46.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:59:46.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:59:46.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.8.29.131:0
07:59:46.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
07:59:46.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:59:46.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:59:46.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/data/info.pb
07:59:46.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.8.29.131
07:59:46.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
07:59:46.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.8.29.190:44257,127.8.29.189:39965,127.8.29.188:37745
07:59:46.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
07:59:46.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:59:46.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
07:59:46.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
07:59:46.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:59:46.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.8.29.131
07:59:46.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/logs
07:59:46.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:59:46.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
07:59:46.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
07:59:46.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
07:59:46.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
07:59:46.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
07:59:46.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
07:59:46.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 07:43:24 UTC on bdcb31816ec0
07:59:46.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11954
07:59:46.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
07:59:46.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.804414  8863 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
07:59:46.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.806438  8863 file_cache.cc:492] Constructed file cache file cache with capacity 419430
07:59:46.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:46.816658  8868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:59:46.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:46.816742  8869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:59:46.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:46.817302  8871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:59:46.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.819164  8863 server_base.cc:1061] running on GCE node
07:59:46.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.819792  8863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
07:59:46.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.820775  8863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
07:59:46.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.822000  8863 hybrid_clock.cc:648] HybridClock initialized: now 1779091186821929 us; error 61 us; skew 500 ppm
07:59:46.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.822302  8863 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
07:59:46.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.824666  8863 webserver.cc:533] Webserver started at http://127.8.29.131:42487/ using document root <none> and password file <none>
07:59:46.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.825412  8863 fs_manager.cc:362] Metadata directory not provided
07:59:46.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.825528  8863 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
07:59:46.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.825924  8863 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
07:59:46.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.828485  8863 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/data/instance:
07:59:46.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "44002126d86d4b42930c65d831be6728"
07:59:46.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 07:59:46 on dist-test-slave-c1kb"
07:59:46.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.829329  8863 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/wal/instance:
07:59:46.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "44002126d86d4b42930c65d831be6728"
07:59:46.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 07:59:46 on dist-test-slave-c1kb"
07:59:46.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.834887  8863 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
07:59:46.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.838672  8877 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
07:59:46.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.840260  8863 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
07:59:46.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.840410  8863 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/data,/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/wal
07:59:46.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "44002126d86d4b42930c65d831be6728"
07:59:46.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 07:59:46 on dist-test-slave-c1kb"
07:59:46.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.840576  8863 fs_report.cc:389] FS layout report
07:59:46.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
07:59:46.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/wal
07:59:46.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/wal
07:59:46.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/data/data
07:59:46.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
07:59:46.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
07:59:46.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
07:59:46.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
07:59:46.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
07:59:46.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
07:59:46.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
07:59:46.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
07:59:46.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
07:59:46.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
07:59:46.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
07:59:46.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.876482  8863 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
07:59:46.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.877328  8863 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
07:59:46.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.877586  8863 kserver.cc:163] Server-wide thread pool size limit: 3276
07:59:46.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.878540  8863 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
07:59:46.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.880360  8863 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
07:59:46.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.880450  8863 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
07:59:46.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.880522  8863 ts_tablet_manager.cc:616] Registered 0 tablets
07:59:46.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.880573  8863 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
07:59:46.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.916877  8863 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.131:37185
07:59:46.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.917004  8989 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.29.131:37185 every 8 connection(s)
07:59:46.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.918846  8863 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/data/info.pb
07:59:46.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.925746  8310 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskivGXg8/build/asan/bin/kudu as pid 8863
07:59:46.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.925937  8310 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/wal/instance
07:59:46.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.940616  8991 heartbeater.cc:344] Connected to a master server at 127.8.29.190:44257
07:59:46.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.941023  8991 heartbeater.cc:461] Registering TS with master...
07:59:46.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.942034  8991 heartbeater.cc:507] Master 127.8.29.190:44257 requested a full tablet report, sending...
07:59:46.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.943194  8992 heartbeater.cc:344] Connected to a master server at 127.8.29.189:39965
07:59:46.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.943318  8992 heartbeater.cc:461] Registering TS with master...
07:59:46.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.943881  8992 heartbeater.cc:507] Master 127.8.29.189:39965 requested a full tablet report, sending...
07:59:46.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.943998  8353 ts_manager.cc:194] Registered new tserver with Master: 44002126d86d4b42930c65d831be6728 (127.8.29.131:37185)
07:59:46.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.945358  8424 ts_manager.cc:194] Registered new tserver with Master: 44002126d86d4b42930c65d831be6728 (127.8.29.131:37185)
07:59:46.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.945612  8990 heartbeater.cc:344] Connected to a master server at 127.8.29.188:37745
07:59:46.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.945739  8990 heartbeater.cc:461] Registering TS with master...
07:59:46.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.946296  8990 heartbeater.cc:507] Master 127.8.29.188:37745 requested a full tablet report, sending...
07:59:46.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.947185  8495 ts_manager.cc:194] Registered new tserver with Master: 44002126d86d4b42930c65d831be6728 (127.8.29.131:37185)
07:59:46.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.948354  8495 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.8.29.131:59125
07:59:46.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.955075  8310 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
07:59:46.957 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
07:59:46.958 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_masters {
}

07:59:46.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:46.968016  8719 heartbeater.cc:499] Master 127.8.29.188:37745 was elected leader, sending a full tablet report...
07:59:46.969 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_masters {
  masters {
    id {
      type: MASTER
      index: 0
    }
    bound_rpc_address {
      host: "127.8.29.190"
      port: 44257
    }
    bound_http_address {
      host: "127.8.29.190"
      port: 44149
    }
  }
  masters {
    id {
      type: MASTER
      index: 1
    }
    bound_rpc_address {
      host: "127.8.29.189"
      port: 39965
    }
    bound_http_address {
      host: "127.8.29.189"
      port: 43825
    }
  }
  masters {
    id {
      type: MASTER
      index: 2
    }
    bound_rpc_address {
      host: "127.8.29.188"
      port: 37745
    }
    bound_http_address {
      host: "127.8.29.188"
      port: 33655
    }
  }
}

07:59:46.973 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_tservers {
}

07:59:46.976 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_tservers {
  tservers {
    id {
      type: TSERVER
      index: 0
    }
    bound_rpc_address {
      host: "127.8.29.129"
      port: 44331
    }
    bound_http_address {
      host: "127.8.29.129"
      port: 38159
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 1
    }
    bound_rpc_address {
      host: "127.8.29.130"
      port: 37019
    }
    bound_http_address {
      host: "127.8.29.130"
      port: 44565
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 2
    }
    bound_rpc_address {
      host: "127.8.29.131"
      port: 37185
    }
    bound_http_address {
      host: "127.8.29.131"
      port: 42487
    }
  }
}

07:59:46.976 [INFO - main] (KuduTestHarness.java:157) Creating a new Kudu client...
07:59:47.160 [DEBUG - main] (CompositeStrategy.java:59) Unable to load modern Subject methods, relying only on legacy methods
java.lang.NoSuchMethodException: javax.security.auth.Subject.current()
	at java.base/java.lang.Class.getDeclaredMethod(Class.java:2677)
	at org.apache.kudu.client.internals.ModernStrategy.<init>(ModernStrategy.java:51)
	at org.apache.kudu.client.internals.CompositeStrategy.<init>(CompositeStrategy.java:52)
	at org.apache.kudu.client.internals.CompositeStrategy.<clinit>(CompositeStrategy.java:41)
	at org.apache.kudu.client.internals.SecurityManagerCompatibility.get(SecurityManagerCompatibility.java:43)
	at org.apache.kudu.client.SecurityContext.setupSubject(SecurityContext.java:155)
	at org.apache.kudu.client.SecurityContext.<init>(SecurityContext.java:138)
	at org.apache.kudu.client.AsyncKuduClient.<init>(AsyncKuduClient.java:431)
	at org.apache.kudu.client.AsyncKuduClient$AsyncKuduClientBuilder.build(AsyncKuduClient.java:3109)
	at org.apache.kudu.test.KuduTestHarness.before(KuduTestHarness.java:161)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:50)
	at org.apache.kudu.test.junit.RetryRule$RetryStatement.doOneAttemptAndReport(RetryRule.java:184)
	at org.apache.kudu.test.junit.RetryRule$RetryStatement.evaluate(RetryRule.java:244)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:27)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
	at org.junit.runner.JUnitCore.runMain(JUnitCore.java:77)
	at org.junit.runner.JUnitCore.main(JUnitCore.java:36)
07:59:47.219 [DEBUG - main] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 
07:59:47.376 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.8.29.188:37745(127.8.29.188:37745)] connecting to peer
07:59:47.434 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.8.29.190:44257(127.8.29.190:44257)] connecting to peer
07:59:47.445 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.8.29.189:39965(127.8.29.189:39965)] connecting to peer
07:59:47.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:47.450667  8856 heartbeater.cc:499] Master 127.8.29.188:37745 was elected leader, sending a full tablet report...
07:59:47.457 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.8.29.188:37745(127.8.29.188:37745)] Successfully connected to peer
07:59:47.458 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.8.29.190:44257(127.8.29.190:44257)] Successfully connected to peer
07:59:47.458 [DEBUG - kudu-nio-2] (Connection.java:757) [peer master-127.8.29.189:39965(127.8.29.189:39965)] Successfully connected to peer
07:59:47.556 [DEBUG - kudu-nio-0] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.8.29.188
07:59:47.556 [DEBUG - kudu-nio-1] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.8.29.190
07:59:47.556 [DEBUG - kudu-nio-2] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.8.29.189
07:59:47.785 [DEBUG - kudu-nio-2] (Negotiator.java:919) Authenticated connection [id: 0xd994edd2, L:/127.0.0.1:43680 - R:/127.8.29.189:39965] using SASL/PLAIN
07:59:47.785 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0xeca6ff86, L:/127.0.0.1:44646 - R:/127.8.29.188:37745] using SASL/PLAIN
07:59:47.785 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x6619d64b, L:/127.0.0.1:33034 - R:/127.8.29.190:44257] using SASL/PLAIN
07:59:47.819 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet Kudu Master for table 'Kudu Master' with partition [<start>, <end>)
07:59:47.821 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=CreateTable, tablet=null, attempt=1, TimeoutTracker(timeout=50000, elapsed=480), Traces: [0ms] refreshing cache from master, [22ms] Sub RPC ConnectToMaster: sending RPC to server master-127.8.29.188:37745, [90ms] Sub RPC ConnectToMaster: sending RPC to server master-127.8.29.190:44257, [101ms] Sub RPC ConnectToMaster: sending RPC to server master-127.8.29.189:39965, [453ms] Sub RPC ConnectToMaster: received response from server master-127.8.29.190:44257: OK, [453ms] Sub RPC ConnectToMaster: received response from server master-127.8.29.189:39965: OK, [454ms] Sub RPC ConnectToMaster: received response from server master-127.8.29.188:37745: OK, deferred=null) after lookup
07:59:47.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:47.869339  8495 catalog_manager.cc:2282] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44646:
07:59:47.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) name: "org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328"
07:59:47.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) schema {
07:59:47.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
07:59:47.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "key"
07:59:47.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
07:59:47.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: true
07:59:47.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
07:59:47.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
07:59:47.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
07:59:47.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
07:59:47.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
07:59:47.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
07:59:47.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column1_i"
07:59:47.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
07:59:47.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
07:59:47.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
07:59:47.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
07:59:47.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
07:59:47.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
07:59:47.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
07:59:47.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
07:59:47.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column2_i"
07:59:47.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
07:59:47.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
07:59:47.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
07:59:47.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
07:59:47.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
07:59:47.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
07:59:47.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
07:59:47.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
07:59:47.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column3_s"
07:59:47.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: STRING
07:59:47.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
07:59:47.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: true
07:59:47.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     encoding: DICT_ENCODING
07:59:47.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     compression: LZ4
07:59:47.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 4096
07:59:47.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
07:59:47.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
07:59:47.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
07:59:47.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
07:59:47.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column4_b"
07:59:47.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: BOOL
07:59:47.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
07:59:47.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
07:59:47.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
07:59:47.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
07:59:47.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
07:59:47.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
07:59:47.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
07:59:47.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) partition_schema {
07:59:47.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   range_schema {
07:59:47.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     columns {
07:59:47.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)       name: "key"
07:59:47.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     }
07:59:47.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
07:59:47.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
07:59:47.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:47.871160  8495 catalog_manager.cc:7058] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
07:59:47.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:47.938253  8791 tablet_service.cc:1511] Processing CreateTablet for tablet e1d28f54265f409eb6d7b87d41754485 (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328 [id=1452334ea30a48adaa267b235c774dee]), partition=RANGE (key) PARTITION UNBOUNDED
07:59:47.941 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:894) Opened table 1452334ea30a48adaa267b235c774dee
07:59:47.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:47.939851  8791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1d28f54265f409eb6d7b87d41754485. 1 dirs total, 0 dirs full, 0 dirs failed
07:59:47.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:47.938686  8654 tablet_service.cc:1511] Processing CreateTablet for tablet e1d28f54265f409eb6d7b87d41754485 (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328 [id=1452334ea30a48adaa267b235c774dee]), partition=RANGE (key) PARTITION UNBOUNDED
07:59:47.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:47.945609  8654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1d28f54265f409eb6d7b87d41754485. 1 dirs total, 0 dirs full, 0 dirs failed
07:59:47.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:47.951150  8990 heartbeater.cc:499] Master 127.8.29.188:37745 was elected leader, sending a full tablet report...
07:59:47.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:47.950405  8925 tablet_service.cc:1511] Processing CreateTablet for tablet e1d28f54265f409eb6d7b87d41754485 (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328 [id=1452334ea30a48adaa267b235c774dee]), partition=RANGE (key) PARTITION UNBOUNDED
07:59:47.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:47.954756  8925 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1d28f54265f409eb6d7b87d41754485. 1 dirs total, 0 dirs full, 0 dirs failed
07:59:47.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:47.968668  9023 tablet_bootstrap.cc:492] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393: Bootstrap starting.
07:59:47.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:47.982466  9024 tablet_bootstrap.cc:492] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c: Bootstrap starting.
07:59:47.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:47.984463  9025 tablet_bootstrap.cc:492] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728: Bootstrap starting.
07:59:47.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:47.987056  9023 tablet_bootstrap.cc:654] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393: Neither blocks nor log segments found. Creating new log.
07:59:47.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:47.987373  9025 tablet_bootstrap.cc:654] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728: Neither blocks nor log segments found. Creating new log.
07:59:47.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:47.988315  9023 log.cc:826] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393: Log is configured to *not* fsync() on all Append() calls
07:59:47.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:47.989037  9025 log.cc:826] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728: Log is configured to *not* fsync() on all Append() calls
07:59:47.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:47.990088  9024 tablet_bootstrap.cc:654] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c: Neither blocks nor log segments found. Creating new log.
07:59:47.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:47.991176  9023 tablet_bootstrap.cc:492] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393: No bootstrap required, opened a new log
07:59:47.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:47.991500  9023 ts_tablet_manager.cc:1403] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393: Time spent bootstrapping tablet: real 0.023s	user 0.008s	sys 0.000s
07:59:47.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:47.991456  9024 log.cc:826] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c: Log is configured to *not* fsync() on all Append() calls
07:59:47.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:47.992195  9025 tablet_bootstrap.cc:492] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728: No bootstrap required, opened a new log
07:59:47.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:47.992620  9025 ts_tablet_manager.cc:1403] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728: Time spent bootstrapping tablet: real 0.008s	user 0.007s	sys 0.000s
07:59:47.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:47.994608  9024 tablet_bootstrap.cc:492] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c: No bootstrap required, opened a new log
07:59:47.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:47.995014  9024 ts_tablet_manager.cc:1403] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c: Time spent bootstrapping tablet: real 0.013s	user 0.004s	sys 0.004s
07:59:47.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:47.998836  9023 raft_consensus.cc:359] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
07:59:47.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:47.999599  9023 raft_consensus.cc:385] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
07:59:48.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:47.999878  9023 raft_consensus.cc:740] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcfe780000f44bb5ba6ab6347414c393, State: Initialized, Role: FOLLOWER
07:59:48.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:48.000873  9023 consensus_queue.cc:260] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
07:59:48.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:48.000759  9025 raft_consensus.cc:359] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
07:59:48.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:48.001173  9025 raft_consensus.cc:385] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
07:59:48.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:48.001281  9025 raft_consensus.cc:740] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44002126d86d4b42930c65d831be6728, State: Initialized, Role: FOLLOWER
07:59:48.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:48.002039  9025 consensus_queue.cc:260] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
07:59:48.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:48.003391  9024 raft_consensus.cc:359] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
07:59:48.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:48.003686  9023 ts_tablet_manager.cc:1434] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393: Time spent starting tablet: real 0.012s	user 0.011s	sys 0.000s
07:59:48.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:48.007477  9025 ts_tablet_manager.cc:1434] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728: Time spent starting tablet: real 0.015s	user 0.010s	sys 0.003s
07:59:48.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:48.012842  9024 raft_consensus.cc:385] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
07:59:48.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:48.013510  9024 raft_consensus.cc:740] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21124dcbd9914973a616d9ed69870d0c, State: Initialized, Role: FOLLOWER
07:59:48.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:48.014544  9024 consensus_queue.cc:260] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
07:59:48.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:48.017571  9024 ts_tablet_manager.cc:1434] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c: Time spent starting tablet: real 0.022s	user 0.008s	sys 0.003s
07:59:48.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:48.059793  9031 raft_consensus.cc:493] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
07:59:48.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:48.060060  9031 raft_consensus.cc:515] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
07:59:48.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:48.061866  9031 leader_election.cc:290] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcfe780000f44bb5ba6ab6347414c393 (127.8.29.129:44331), 44002126d86d4b42930c65d831be6728 (127.8.29.131:37185)
07:59:48.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:48.071089  8674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1d28f54265f409eb6d7b87d41754485" candidate_uuid: "21124dcbd9914973a616d9ed69870d0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcfe780000f44bb5ba6ab6347414c393" is_pre_election: true
07:59:48.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:48.071203  8945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1d28f54265f409eb6d7b87d41754485" candidate_uuid: "21124dcbd9914973a616d9ed69870d0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44002126d86d4b42930c65d831be6728" is_pre_election: true
07:59:48.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:48.071705  8945 raft_consensus.cc:2468] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21124dcbd9914973a616d9ed69870d0c in term 0.
07:59:48.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:48.071731  8674 raft_consensus.cc:2468] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21124dcbd9914973a616d9ed69870d0c in term 0.
07:59:48.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:48.072430  8745 leader_election.cc:304] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 21124dcbd9914973a616d9ed69870d0c, 44002126d86d4b42930c65d831be6728, bcfe780000f44bb5ba6ab6347414c393; no voters: 
07:59:48.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:48.072860  9031 raft_consensus.cc:2804] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 0 FOLLOWER]: Leader pre-election won for term 1
07:59:48.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:48.073009  9031 raft_consensus.cc:493] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
07:59:48.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:48.073112  9031 raft_consensus.cc:3060] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 0 FOLLOWER]: Advancing to term 1
07:59:48.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:48.075152  9031 raft_consensus.cc:515] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
07:59:48.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:48.075930  9031 leader_election.cc:290] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [CANDIDATE]: Term 1 election: Requested vote from peers bcfe780000f44bb5ba6ab6347414c393 (127.8.29.129:44331), 44002126d86d4b42930c65d831be6728 (127.8.29.131:37185)
07:59:48.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:48.076399  8674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1d28f54265f409eb6d7b87d41754485" candidate_uuid: "21124dcbd9914973a616d9ed69870d0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcfe780000f44bb5ba6ab6347414c393"
07:59:48.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:48.076575  8674 raft_consensus.cc:3060] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 0 FOLLOWER]: Advancing to term 1
07:59:48.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:48.076711  8945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1d28f54265f409eb6d7b87d41754485" candidate_uuid: "21124dcbd9914973a616d9ed69870d0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44002126d86d4b42930c65d831be6728"
07:59:48.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:48.076932  8945 raft_consensus.cc:3060] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 0 FOLLOWER]: Advancing to term 1
07:59:48.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:48.078883  8674 raft_consensus.cc:2468] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21124dcbd9914973a616d9ed69870d0c in term 1.
07:59:48.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:48.079105  8945 raft_consensus.cc:2468] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21124dcbd9914973a616d9ed69870d0c in term 1.
07:59:48.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:48.079352  8747 leader_election.cc:304] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21124dcbd9914973a616d9ed69870d0c, bcfe780000f44bb5ba6ab6347414c393; no voters: 
07:59:48.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:48.079726  9031 raft_consensus.cc:2804] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 1 FOLLOWER]: Leader election won for term 1
07:59:48.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:48.080276  9031 raft_consensus.cc:697] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 1 LEADER]: Becoming Leader. State: Replica: 21124dcbd9914973a616d9ed69870d0c, State: Running, Role: LEADER
07:59:48.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:48.080875  9031 consensus_queue.cc:237] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
07:59:48.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:48.088007  8495 catalog_manager.cc:5696] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c reported cstate change: term changed from 0 to 1, leader changed from <none> to 21124dcbd9914973a616d9ed69870d0c (127.8.29.130). New cstate: current_term: 1 leader_uuid: "21124dcbd9914973a616d9ed69870d0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } health_report { overall_health: UNKNOWN } } }
07:59:48.149 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:894) Opened table 1452334ea30a48adaa267b235c774dee
07:59:48.167 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
07:59:48.173 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299994, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)[L],44002126d86d4b42930c65d831be6728(127.8.29.131:37185),bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)]}]
07:59:48.174 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=null, attempt=1, sequence_id=-1, TimeoutTracker(timeout=30000, elapsed=17), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [8ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=null) after lookup
07:59:48.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:48.175573  8722 tablet.cc:2404] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393: Can't schedule compaction. Clean time has not been advanced past its initial value.
07:59:48.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:48.179528  8993 tablet.cc:2404] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728: Can't schedule compaction. Clean time has not been advanced past its initial value.
07:59:48.183 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] connecting to peer
07:59:48.186 [DEBUG - kudu-nio-3] (Connection.java:757) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] Successfully connected to peer
07:59:48.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:48.186498  8861 tablet.cc:2404] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
07:59:48.255 [DEBUG - kudu-nio-3] (Negotiator.java:919) Authenticated connection [id: 0x5d4c2903, L:/127.0.0.1:59438 - R:/127.8.29.130:37019] using TOKEN/null
07:59:48.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:48.267738  8945 raft_consensus.cc:1275] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 1 FOLLOWER]: Refusing update from remote peer 21124dcbd9914973a616d9ed69870d0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
07:59:48.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:48.267798  8674 raft_consensus.cc:1275] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 1 FOLLOWER]: Refusing update from remote peer 21124dcbd9914973a616d9ed69870d0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
07:59:48.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:48.268985  9031 consensus_queue.cc:1048] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
07:59:48.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:48.269323  9034 consensus_queue.cc:1048] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
07:59:48.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:48.310990  9038 mvcc.cc:204] Tried to move back new op lower bound from 7287157507132436480 to 7287157506383343616. Current Snapshot: MvccSnapshot[applied={T|T < 7287157507132436480}]
07:59:48.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:48.313509  9040 mvcc.cc:204] Tried to move back new op lower bound from 7287157507132436480 to 7287157506383343616. Current Snapshot: MvccSnapshot[applied={T|T < 7287157507132436480}]
07:59:48.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:48.328286  9039 mvcc.cc:204] Tried to move back new op lower bound from 7287157507132436480 to 7287157506383343616. Current Snapshot: MvccSnapshot[applied={T|T < 7287157507132436480}]
07:59:48.450 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 3 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328, tablet=e1d28f54265f409eb6d7b87d41754485, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
07:59:48.452 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.8.29.130:37019
07:59:48.454 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

07:59:48.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:48.454545  8310 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskivGXg8/build/asan/bin/kudu pid 8729
07:59:48.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:48.493129  8851 generic_service.cc:196] Checking for leaks (request via RPC)
07:59:48.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:48.598795  8310 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskivGXg8/build/asan/bin/kudu with pid 8729
07:59:48.618 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
07:59:48.621 [DEBUG - kudu-nio-3] (Connection.java:239) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] handling channelInactive
07:59:48.621 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] cleaning up while in state READY due to: connection closed
07:59:48.621 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019) for tablet e1d28f54265f409eb6d7b87d41754485: connection closed
07:59:48.645 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
07:59:48.645 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)[L],44002126d86d4b42930c65d831be6728(127.8.29.131:37185),bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)]}]
07:59:48.646 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=2, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=27), Traces: [0ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [3ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [27ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@1819806159(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@31b9735f -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@113ee9d -> wakeup thread Time-limited test)) after lookup
07:59:48.647 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] connecting to peer
07:59:48.649 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
07:59:48.649 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] cleaning up while in state CONNECTING due to: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
07:59:48.649 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
07:59:48.666 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
07:59:48.666 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)[L],44002126d86d4b42930c65d831be6728(127.8.29.131:37185),bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)]}]
07:59:48.667 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=4, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=48), Traces: [0ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [3ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [27ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [28ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [31ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [31ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [44ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [48ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@1819806159(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@31b9735f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1919795946 after retry RPC@1914355625), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@113ee9d -> wakeup thread Time-limited test -> (continuation of Deferred@1919795946 after retry RPC@1914355625))) after lookup
07:59:48.668 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] connecting to peer
07:59:48.670 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
07:59:48.670 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] cleaning up while in state CONNECTING due to: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
07:59:48.670 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
07:59:48.686 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
07:59:48.686 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)[L],44002126d86d4b42930c65d831be6728(127.8.29.131:37185),bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)]}]
07:59:48.687 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=6, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=67), Traces: [0ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [3ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [27ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [28ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [31ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [31ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [44ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [48ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [49ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [52ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [52ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [67ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@1819806159(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@31b9735f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1919795946 after retry RPC@1914355625) -> (continuation of Deferred@2114060318 after retry RPC@59430145), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@113ee9d -> wakeup thread Time-limited test -> (continuation of Deferred@1919795946 after retry RPC@1914355625) -> (continuation of Deferred@2114060318 after retry RPC@59430145))) after lookup
07:59:48.688 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] connecting to peer
07:59:48.690 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
07:59:48.690 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] cleaning up while in state CONNECTING due to: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
07:59:48.690 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
07:59:48.766 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
07:59:48.766 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)[L],44002126d86d4b42930c65d831be6728(127.8.29.131:37185),bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)]}]
07:59:48.767 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=8, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=148), Traces: [0ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [3ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [27ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [28ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [31ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [31ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [44ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [48ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [49ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [52ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [52ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [67ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [69ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [72ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [72ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [144ms] refreshing cache from master, [145ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [148ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@1819806159(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@31b9735f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1919795946 after retry RPC@1914355625) -> (continuation of Deferred@2114060318 after retry RPC@59430145) -> (continuation of Deferred@333068657 after retry RPC@2040334602), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@113ee9d -> wakeup thread Time-limited test -> (continuation of Deferred@1919795946 after retry RPC@1914355625) -> (continuation of Deferred@2114060318 after retry RPC@59430145) -> (continuation of Deferred@333068657 after retry RPC@2040334602))) after lookup
07:59:48.767 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] connecting to peer
07:59:48.769 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
07:59:48.769 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] cleaning up while in state CONNECTING due to: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
07:59:48.769 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
07:59:48.806 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
07:59:48.807 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)[L],44002126d86d4b42930c65d831be6728(127.8.29.131:37185),bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)]}]
07:59:48.808 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=10, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=189), Traces: [0ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [3ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [27ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [28ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [31ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [31ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [44ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [48ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [49ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [52ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [52ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [67ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [69ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [72ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [72ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [144ms] refreshing cache from master, [145ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [148ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [149ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [151ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [151ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [184ms] refreshing cache from master, [184ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [188ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@1819806159(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@31b9735f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1919795946 after retry RPC@1914355625) -> (continuation of Deferred@2114060318 after retry RPC@59430145) -> (continuation of Deferred@333068657 after retry RPC@2040334602) -> (continuation of Deferred@600878551 after retry RPC@1965075825), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@113ee9d -> wakeup thread Time-limited test -> (continuation of Deferred@1919795946 after retry RPC@1914355625) -> (continuation of Deferred@2114060318 after retry RPC@59430145) -> (continuation of Deferred@333068657 after retry RPC@2040334602) -> (continuation of Deferred@600878551 after retry RPC@1965075825))) after lookup
07:59:48.808 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] connecting to peer
07:59:48.809 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
07:59:48.810 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] cleaning up while in state CONNECTING due to: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
07:59:48.810 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
07:59:49.566 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
07:59:49.566 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)[L],44002126d86d4b42930c65d831be6728(127.8.29.131:37185),bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)]}]
07:59:49.567 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=12, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=948), Traces: [0ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [3ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [27ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [28ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [31ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [31ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [44ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [48ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [49ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [52ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [52ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [67ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [69ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [72ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [72ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [144ms] refreshing cache from master, [145ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [148ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [149ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [151ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [151ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [184ms] refreshing cache from master, [184ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [188ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [190ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [192ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [192ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [944ms] refreshing cache from master, [945ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [948ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@1819806159(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@31b9735f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1919795946 after retry RPC@1914355625) -> (continuation of Deferred@2114060318 after retry RPC@59430145) -> (continuation of Deferred@333068657 after retry RPC@2040334602) -> (continuation of Deferred@600878551 after retry RPC@1965075825) -> (continuation of Deferred@1536207586 after retry RPC@650970696), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@113ee9d -> wakeup thread Time-limited test -> (continuation of Deferred@1919795946 after retry RPC@1914355625) -> (continuation of Deferred@2114060318 after retry RPC@59430145) -> (continuation of Deferred@333068657 after retry RPC@2040334602) -> (continuation of Deferred@600878551 after retry RPC@1965075825) -> (continuation of Deferred@1536207586 after retry RPC@650970696))) after lookup
07:59:49.568 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] connecting to peer
07:59:49.569 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
07:59:49.569 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] cleaning up while in state CONNECTING due to: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
07:59:49.569 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
07:59:49.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:49.876202  9054 raft_consensus.cc:493] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 21124dcbd9914973a616d9ed69870d0c)
07:59:49.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:49.876511  9054 raft_consensus.cc:515] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
07:59:49.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:49.878230  9054 leader_election.cc:290] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcfe780000f44bb5ba6ab6347414c393 (127.8.29.129:44331), 21124dcbd9914973a616d9ed69870d0c (127.8.29.130:37019)
07:59:49.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:49.879561  8878 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.29.130:37019: connect: Connection refused (error 111)
07:59:49.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:49.881428  8878 leader_election.cc:336] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 21124dcbd9914973a616d9ed69870d0c (127.8.29.130:37019): Network error: Client connection negotiation failed: client connection to 127.8.29.130:37019: connect: Connection refused (error 111)
07:59:49.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:49.885890  8673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1d28f54265f409eb6d7b87d41754485" candidate_uuid: "44002126d86d4b42930c65d831be6728" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bcfe780000f44bb5ba6ab6347414c393" is_pre_election: true
07:59:49.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:49.886656  8881 leader_election.cc:304] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44002126d86d4b42930c65d831be6728; no voters: 21124dcbd9914973a616d9ed69870d0c, bcfe780000f44bb5ba6ab6347414c393
07:59:49.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:49.887044  9054 raft_consensus.cc:2749] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
07:59:49.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:49.889693  9059 raft_consensus.cc:493] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 21124dcbd9914973a616d9ed69870d0c)
07:59:49.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:49.889848  9059 raft_consensus.cc:515] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
07:59:49.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:49.891244  9059 leader_election.cc:290] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 21124dcbd9914973a616d9ed69870d0c (127.8.29.130:37019), 44002126d86d4b42930c65d831be6728 (127.8.29.131:37185)
07:59:49.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:49.892611  8607 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.29.130:37019: connect: Connection refused (error 111)
07:59:49.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:49.894289  8607 leader_election.cc:336] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 21124dcbd9914973a616d9ed69870d0c (127.8.29.130:37019): Network error: Client connection negotiation failed: client connection to 127.8.29.130:37019: connect: Connection refused (error 111)
07:59:49.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:49.899518  8945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1d28f54265f409eb6d7b87d41754485" candidate_uuid: "bcfe780000f44bb5ba6ab6347414c393" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "44002126d86d4b42930c65d831be6728" is_pre_election: true
07:59:49.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:49.899734  8945 raft_consensus.cc:2468] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcfe780000f44bb5ba6ab6347414c393 in term 1.
07:59:49.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:49.900305  8608 leader_election.cc:304] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44002126d86d4b42930c65d831be6728, bcfe780000f44bb5ba6ab6347414c393; no voters: 21124dcbd9914973a616d9ed69870d0c
07:59:49.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:49.900736  9059 raft_consensus.cc:2804] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 1 FOLLOWER]: Leader pre-election won for term 2
07:59:49.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:49.900888  9059 raft_consensus.cc:493] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 21124dcbd9914973a616d9ed69870d0c)
07:59:49.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:49.900997  9059 raft_consensus.cc:3060] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 1 FOLLOWER]: Advancing to term 2
07:59:49.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:49.903039  9059 raft_consensus.cc:515] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
07:59:49.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:49.903833  9059 leader_election.cc:290] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [CANDIDATE]: Term 2 election: Requested vote from peers 21124dcbd9914973a616d9ed69870d0c (127.8.29.130:37019), 44002126d86d4b42930c65d831be6728 (127.8.29.131:37185)
07:59:49.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:49.904541  8945 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1d28f54265f409eb6d7b87d41754485" candidate_uuid: "bcfe780000f44bb5ba6ab6347414c393" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "44002126d86d4b42930c65d831be6728"
07:59:49.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:49.904770  8945 raft_consensus.cc:3060] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 1 FOLLOWER]: Advancing to term 2
07:59:49.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:49.905474  8607 leader_election.cc:336] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 21124dcbd9914973a616d9ed69870d0c (127.8.29.130:37019): Network error: Client connection negotiation failed: client connection to 127.8.29.130:37019: connect: Connection refused (error 111)
07:59:49.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:49.907069  8945 raft_consensus.cc:2468] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcfe780000f44bb5ba6ab6347414c393 in term 2.
07:59:49.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:49.907549  8608 leader_election.cc:304] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44002126d86d4b42930c65d831be6728, bcfe780000f44bb5ba6ab6347414c393; no voters: 21124dcbd9914973a616d9ed69870d0c
07:59:49.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:49.907859  9059 raft_consensus.cc:2804] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 2 FOLLOWER]: Leader election won for term 2
07:59:49.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:49.908434  9059 raft_consensus.cc:697] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 2 LEADER]: Becoming Leader. State: Replica: bcfe780000f44bb5ba6ab6347414c393, State: Running, Role: LEADER
07:59:49.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:49.908988  9059 consensus_queue.cc:237] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
07:59:49.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:49.913260  8495 catalog_manager.cc:5696] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 reported cstate change: term changed from 1 to 2, leader changed from 21124dcbd9914973a616d9ed69870d0c (127.8.29.130) to bcfe780000f44bb5ba6ab6347414c393 (127.8.29.129). New cstate: current_term: 2 leader_uuid: "bcfe780000f44bb5ba6ab6347414c393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } health_report { overall_health: UNKNOWN } } }
07:59:50.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:50.286794  8945 raft_consensus.cc:1275] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 2 FOLLOWER]: Refusing update from remote peer bcfe780000f44bb5ba6ab6347414c393: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
07:59:50.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:50.287604  9059 consensus_queue.cc:1048] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
07:59:50.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:50.294696  8607 consensus_peers.cc:597] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 -> Peer 21124dcbd9914973a616d9ed69870d0c (127.8.29.130:37019): Couldn't send request to peer 21124dcbd9914973a616d9ed69870d0c. Status: Network error: Client connection negotiation failed: client connection to 127.8.29.130:37019: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
07:59:52.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:52.698450  8607 consensus_peers.cc:597] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 -> Peer 21124dcbd9914973a616d9ed69870d0c (127.8.29.130:37019): Couldn't send request to peer 21124dcbd9914973a616d9ed69870d0c. Status: Network error: Client connection negotiation failed: client connection to 127.8.29.130:37019: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
07:59:52.946 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
07:59:52.947 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019),44002126d86d4b42930c65d831be6728(127.8.29.131:37185),bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)[L]]}]
07:59:52.948 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=14, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=4328), Traces: [0ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [3ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [27ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [28ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [31ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [31ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [44ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [48ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [49ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [52ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [52ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [67ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [69ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [72ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [72ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [144ms] refreshing cache from master, [145ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [148ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [149ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [151ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [151ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [184ms] refreshing cache from master, [184ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [188ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [190ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [192ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [192ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [944ms] refreshing cache from master, [945ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [948ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [950ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [951ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [951ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [4324ms] refreshing cache from master, [4324ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [4328ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@1819806159(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@31b9735f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1919795946 after retry RPC@1914355625) -> (continuation of Deferred@2114060318 after retry RPC@59430145) -> (continuation of Deferred@333068657 after retry RPC@2040334602) -> (continuation of Deferred@600878551 after retry RPC@1965075825) -> (continuation of Deferred@1536207586 after retry RPC@650970696) -> (continuation of Deferred@1404707645 after retry RPC@495681604), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@113ee9d -> wakeup thread Time-limited test -> (continuation of Deferred@1919795946 after retry RPC@1914355625) -> (continuation of Deferred@2114060318 after retry RPC@59430145) -> (continuation of Deferred@333068657 after retry RPC@2040334602) -> (continuation of Deferred@600878551 after retry RPC@1965075825) -> (continuation of Deferred@1536207586 after retry RPC@650970696) -> (continuation of Deferred@1404707645 after retry RPC@495681604))) after lookup
07:59:52.949 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] connecting to peer
07:59:52.951 [DEBUG - kudu-nio-2] (Connection.java:757) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] Successfully connected to peer
07:59:52.975 [DEBUG - kudu-nio-2] (Negotiator.java:919) Authenticated connection [id: 0xaedc5c45, L:/127.0.0.1:51554 - R:/127.8.29.129:44331] using TOKEN/null
07:59:53.009 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.8.29.130:37019
07:59:53.012 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

07:59:53.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.013103  8310 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskivGXg8/build/asan/bin/kudu
07:59:53.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskivGXg8/build/asan/bin/kudu
07:59:53.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/wal
07:59:53.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/data
07:59:53.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
07:59:53.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
07:59:53.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
07:59:53.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:59:53.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:59:53.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:59:53.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/logs
07:59:53.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/data/info.pb
07:59:53.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:59:53.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
07:59:53.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
07:59:53.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
07:59:53.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
07:59:53.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:59:53.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:59:53.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:59:53.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
07:59:53.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
07:59:53.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.8.29.130:37019
07:59:53.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.8.29.130
07:59:53.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.8.29.190:44257,127.8.29.189:39965,127.8.29.188:37745
07:59:53.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44565
07:59:53.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.8.29.130
07:59:53.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.8.29.148:45029
07:59:53.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:59:53.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:59:53.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
07:59:53.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:53.400579  9093 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
07:59:53.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:53.400940  9093 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
07:59:53.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:53.401053  9093 flags.cc:432] Enabled unsafe flag: --never_fsync=true
07:59:53.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:53.410184  9093 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
07:59:53.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:53.410423  9093 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.130
07:59:53.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.421180  9093 tablet_server_runner.cc:78] Tablet server non-default flags:
07:59:53.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:59:53.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.8.29.148:45029
07:59:53.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:59:53.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:59:53.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/data
07:59:53.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/wal
07:59:53.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:59:53.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:59:53.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.8.29.130:37019
07:59:53.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
07:59:53.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:59:53.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:59:53.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/data/info.pb
07:59:53.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.8.29.130
07:59:53.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44565
07:59:53.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.8.29.190:44257,127.8.29.189:39965,127.8.29.188:37745
07:59:53.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
07:59:53.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:59:53.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
07:59:53.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
07:59:53.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:59:53.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.8.29.130
07:59:53.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/logs
07:59:53.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:59:53.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
07:59:53.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
07:59:53.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
07:59:53.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
07:59:53.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
07:59:53.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
07:59:53.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 07:43:24 UTC on bdcb31816ec0
07:59:53.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11954
07:59:53.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
07:59:53.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.423362  9093 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
07:59:53.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.425330  9093 file_cache.cc:492] Constructed file cache file cache with capacity 419430
07:59:53.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:53.436406  9099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:59:53.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:53.436751  9101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:59:53.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:53.436441  9098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:59:53.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.441385  9093 server_base.cc:1061] running on GCE node
07:59:53.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.442085  9093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
07:59:53.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.443130  9093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
07:59:53.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.444353  9093 hybrid_clock.cc:648] HybridClock initialized: now 1779091193444309 us; error 41 us; skew 500 ppm
07:59:53.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.444700  9093 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
07:59:53.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.447634  9093 webserver.cc:533] Webserver started at http://127.8.29.130:44565/ using document root <none> and password file <none>
07:59:53.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.448352  9093 fs_manager.cc:362] Metadata directory not provided
07:59:53.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.448450  9093 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
07:59:53.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.453867  9093 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.000s
07:59:53.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.457094  9107 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
07:59:53.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.458658  9093 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
07:59:53.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.458789  9093 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/data,/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/wal
07:59:53.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "21124dcbd9914973a616d9ed69870d0c"
07:59:53.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 07:59:46 on dist-test-slave-c1kb"
07:59:53.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.459581  9093 fs_report.cc:389] FS layout report
07:59:53.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
07:59:53.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/wal
07:59:53.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/wal
07:59:53.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/data/data
07:59:53.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
07:59:53.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
07:59:53.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
07:59:53.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
07:59:53.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
07:59:53.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
07:59:53.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
07:59:53.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
07:59:53.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
07:59:53.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
07:59:53.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
07:59:53.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.478721  9093 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
07:59:53.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.479674  9093 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
07:59:53.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.479928  9093 kserver.cc:163] Server-wide thread pool size limit: 3276
07:59:53.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.480909  9093 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
07:59:53.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.483209  9114 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
07:59:53.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.489391  9093 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
07:59:53.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.489483  9093 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.007s	user 0.001s	sys 0.000s
07:59:53.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.489591  9093 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
07:59:53.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.492599  9093 ts_tablet_manager.cc:616] Registered 1 tablets
07:59:53.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.492694  9093 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.002s	sys 0.000s
07:59:53.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.493242  9114 tablet_bootstrap.cc:492] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c: Bootstrap starting.
07:59:53.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.519280  9114 log.cc:826] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c: Log is configured to *not* fsync() on all Append() calls
07:59:53.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.536265  9093 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.130:37019
07:59:53.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.536350  9221 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.29.130:37019 every 8 connection(s)
07:59:53.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.537992  9093 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/data/info.pb
07:59:53.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.538609  9114 tablet_bootstrap.cc:492] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
07:59:53.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.539963  9114 tablet_bootstrap.cc:492] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c: Bootstrap complete.
07:59:53.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.542681  9114 ts_tablet_manager.cc:1403] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c: Time spent bootstrapping tablet: real 0.050s	user 0.042s	sys 0.005s
07:59:53.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.547986  8310 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskivGXg8/build/asan/bin/kudu as pid 9093
07:59:53.548 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
07:59:53.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.551220  9114 raft_consensus.cc:359] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
07:59:53.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.552282  9114 raft_consensus.cc:740] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21124dcbd9914973a616d9ed69870d0c, State: Initialized, Role: FOLLOWER
07:59:53.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.555939  9114 consensus_queue.cc:260] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
07:59:53.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.574030  9114 ts_tablet_manager.cc:1434] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c: Time spent starting tablet: real 0.031s	user 0.008s	sys 0.012s
07:59:53.577 [DEBUG - kudu-nio-2] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 6 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328, tablet=e1d28f54265f409eb6d7b87d41754485, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
07:59:53.578 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.8.29.129:44331
07:59:53.579 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

07:59:53.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.579300  8310 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskivGXg8/build/asan/bin/kudu pid 8582
07:59:53.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.585196  9222 heartbeater.cc:344] Connected to a master server at 127.8.29.188:37745
07:59:53.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.585611  9222 heartbeater.cc:461] Registering TS with master...
07:59:53.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.586587  9222 heartbeater.cc:507] Master 127.8.29.188:37745 requested a full tablet report, sending...
07:59:53.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.587848  9223 heartbeater.cc:344] Connected to a master server at 127.8.29.190:44257
07:59:53.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.588034  9223 heartbeater.cc:461] Registering TS with master...
07:59:53.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.588770  9223 heartbeater.cc:507] Master 127.8.29.190:44257 requested a full tablet report, sending...
07:59:53.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.589906  8494 ts_manager.cc:194] Re-registered known tserver with Master: 21124dcbd9914973a616d9ed69870d0c (127.8.29.130:37019)
07:59:53.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.590397  8352 ts_manager.cc:194] Re-registered known tserver with Master: 21124dcbd9914973a616d9ed69870d0c (127.8.29.130:37019)
07:59:53.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.591513  8494 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.8.29.130:42115
07:59:53.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.594425  9222 heartbeater.cc:499] Master 127.8.29.188:37745 was elected leader, sending a full tablet report...
07:59:53.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.597499  9224 heartbeater.cc:344] Connected to a master server at 127.8.29.189:39965
07:59:53.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.597666  9224 heartbeater.cc:461] Registering TS with master...
07:59:53.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.598008  9224 heartbeater.cc:507] Master 127.8.29.189:39965 requested a full tablet report, sending...
07:59:53.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.599542  8423 ts_manager.cc:194] Re-registered known tserver with Master: 21124dcbd9914973a616d9ed69870d0c (127.8.29.130:37019)
07:59:53.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.602700  8714 generic_service.cc:196] Checking for leaks (request via RPC)
07:59:53.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:53.706200  8310 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskivGXg8/build/asan/bin/kudu with pid 8582
07:59:53.726 [DEBUG - kudu-nio-2] (Connection.java:239) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] handling channelInactive
07:59:53.726 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
07:59:53.726 [DEBUG - kudu-nio-2] (Connection.java:709) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] cleaning up while in state READY due to: connection closed
07:59:53.727 [DEBUG - Time-limited test] (Connection.java:748) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] connecting to peer
07:59:53.728 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
07:59:53.728 [DEBUG - kudu-nio-3] (Connection.java:709) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] cleaning up while in state CONNECTING due to: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
07:59:53.729 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
07:59:53.746 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
07:59:53.746 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019),44002126d86d4b42930c65d831be6728(127.8.29.131:37185),bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)[L]]}]
07:59:53.747 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=2, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=20), Traces: [0ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [2ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [2ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [18ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@1296239173(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@c8b3a09 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@17ee5f5e -> wakeup thread Time-limited test)) after lookup
07:59:53.747 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] connecting to peer
07:59:53.748 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
07:59:53.749 [DEBUG - kudu-nio-4] (Connection.java:709) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] cleaning up while in state CONNECTING due to: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
07:59:53.749 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
07:59:53.766 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
07:59:53.766 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019),44002126d86d4b42930c65d831be6728(127.8.29.131:37185),bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)[L]]}]
07:59:53.766 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=4, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=39), Traces: [0ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [2ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [2ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [18ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [20ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [22ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [22ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [35ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [38ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@1296239173(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@c8b3a09 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1262515787 after retry RPC@783402194), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@17ee5f5e -> wakeup thread Time-limited test -> (continuation of Deferred@1262515787 after retry RPC@783402194))) after lookup
07:59:53.767 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] connecting to peer
07:59:53.768 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
07:59:53.768 [DEBUG - kudu-nio-5] (Connection.java:709) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] cleaning up while in state CONNECTING due to: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
07:59:53.768 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
07:59:53.806 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
07:59:53.806 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019),44002126d86d4b42930c65d831be6728(127.8.29.131:37185),bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)[L]]}]
07:59:53.807 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=6, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=80), Traces: [0ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [2ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [2ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [18ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [20ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [22ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [22ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [35ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [38ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [40ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [41ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [42ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [75ms] refreshing cache from master, [76ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [79ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@1296239173(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@c8b3a09 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1262515787 after retry RPC@783402194) -> (continuation of Deferred@317394661 after retry RPC@844943166), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@17ee5f5e -> wakeup thread Time-limited test -> (continuation of Deferred@1262515787 after retry RPC@783402194) -> (continuation of Deferred@317394661 after retry RPC@844943166))) after lookup
07:59:53.808 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] connecting to peer
07:59:53.809 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
07:59:53.809 [DEBUG - kudu-nio-6] (Connection.java:709) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] cleaning up while in state CONNECTING due to: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
07:59:53.809 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
07:59:53.865 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
07:59:53.865 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019),44002126d86d4b42930c65d831be6728(127.8.29.131:37185),bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)[L]]}]
07:59:53.866 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=8, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=139), Traces: [0ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [2ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [2ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [18ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [20ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [22ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [22ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [35ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [38ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [40ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [41ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [42ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [75ms] refreshing cache from master, [76ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [79ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [80ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [82ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [82ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [135ms] refreshing cache from master, [135ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [138ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@1296239173(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@c8b3a09 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1262515787 after retry RPC@783402194) -> (continuation of Deferred@317394661 after retry RPC@844943166) -> (continuation of Deferred@970405587 after retry RPC@1831222248), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@17ee5f5e -> wakeup thread Time-limited test -> (continuation of Deferred@1262515787 after retry RPC@783402194) -> (continuation of Deferred@317394661 after retry RPC@844943166) -> (continuation of Deferred@970405587 after retry RPC@1831222248))) after lookup
07:59:53.867 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] connecting to peer
07:59:53.869 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
07:59:53.869 [DEBUG - kudu-nio-7] (Connection.java:709) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] cleaning up while in state CONNECTING due to: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
07:59:53.869 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
07:59:54.046 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
07:59:54.046 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019),44002126d86d4b42930c65d831be6728(127.8.29.131:37185),bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)[L]]}]
07:59:54.047 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=10, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=320), Traces: [0ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [2ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [2ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [18ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [20ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [22ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [22ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [35ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [38ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [40ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [41ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [42ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [75ms] refreshing cache from master, [76ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [79ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [80ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [82ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [82ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [135ms] refreshing cache from master, [135ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [138ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [140ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [142ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [142ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [315ms] refreshing cache from master, [315ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [318ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@1296239173(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@c8b3a09 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1262515787 after retry RPC@783402194) -> (continuation of Deferred@317394661 after retry RPC@844943166) -> (continuation of Deferred@970405587 after retry RPC@1831222248) -> (continuation of Deferred@1781049071 after retry RPC@777192796), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@17ee5f5e -> wakeup thread Time-limited test -> (continuation of Deferred@1262515787 after retry RPC@783402194) -> (continuation of Deferred@317394661 after retry RPC@844943166) -> (continuation of Deferred@970405587 after retry RPC@1831222248) -> (continuation of Deferred@1781049071 after retry RPC@777192796))) after lookup
07:59:54.048 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] connecting to peer
07:59:54.049 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
07:59:54.049 [DEBUG - kudu-nio-0] (Connection.java:709) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] cleaning up while in state CONNECTING due to: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
07:59:54.049 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
07:59:54.665 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
07:59:54.665 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019),44002126d86d4b42930c65d831be6728(127.8.29.131:37185),bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)[L]]}]
07:59:54.666 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=12, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=939), Traces: [0ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [2ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [2ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [18ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [20ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [22ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [22ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [35ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [38ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [40ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [41ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [42ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [75ms] refreshing cache from master, [76ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [79ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [80ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [82ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [82ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [135ms] refreshing cache from master, [135ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [138ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [140ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [142ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [142ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [315ms] refreshing cache from master, [315ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [318ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [320ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [322ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [322ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [935ms] refreshing cache from master, [935ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [938ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@1296239173(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@c8b3a09 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1262515787 after retry RPC@783402194) -> (continuation of Deferred@317394661 after retry RPC@844943166) -> (continuation of Deferred@970405587 after retry RPC@1831222248) -> (continuation of Deferred@1781049071 after retry RPC@777192796) -> (continuation of Deferred@1261370003 after retry RPC@265080646), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@17ee5f5e -> wakeup thread Time-limited test -> (continuation of Deferred@1262515787 after retry RPC@783402194) -> (continuation of Deferred@317394661 after retry RPC@844943166) -> (continuation of Deferred@970405587 after retry RPC@1831222248) -> (continuation of Deferred@1781049071 after retry RPC@777192796) -> (continuation of Deferred@1261370003 after retry RPC@265080646))) after lookup
07:59:54.667 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] connecting to peer
07:59:54.668 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
07:59:54.668 [DEBUG - kudu-nio-1] (Connection.java:709) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] cleaning up while in state CONNECTING due to: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
07:59:54.668 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
07:59:55.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:55.096881  9237 raft_consensus.cc:493] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bcfe780000f44bb5ba6ab6347414c393)
07:59:55.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:55.097054  9237 raft_consensus.cc:515] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
07:59:55.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:55.097703  9237 leader_election.cc:290] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bcfe780000f44bb5ba6ab6347414c393 (127.8.29.129:44331), 21124dcbd9914973a616d9ed69870d0c (127.8.29.130:37019)
07:59:55.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:55.098793  8881 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.29.129:44331: connect: Connection refused (error 111)
07:59:55.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:55.100615  8881 leader_election.cc:336] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bcfe780000f44bb5ba6ab6347414c393 (127.8.29.129:44331): Network error: Client connection negotiation failed: client connection to 127.8.29.129:44331: connect: Connection refused (error 111)
07:59:55.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:55.106145  9176 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1d28f54265f409eb6d7b87d41754485" candidate_uuid: "44002126d86d4b42930c65d831be6728" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "21124dcbd9914973a616d9ed69870d0c" is_pre_election: true
07:59:55.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:55.106586  9176 raft_consensus.cc:2468] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44002126d86d4b42930c65d831be6728 in term 1.
07:59:55.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:55.107618  8878 leader_election.cc:304] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 21124dcbd9914973a616d9ed69870d0c, 44002126d86d4b42930c65d831be6728; no voters: bcfe780000f44bb5ba6ab6347414c393
07:59:55.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:55.107944  9237 raft_consensus.cc:2804] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 2 FOLLOWER]: Leader pre-election won for term 3
07:59:55.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:55.108094  9237 raft_consensus.cc:493] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 2 FOLLOWER]: Starting leader election (detected failure of leader bcfe780000f44bb5ba6ab6347414c393)
07:59:55.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:55.108189  9237 raft_consensus.cc:3060] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 2 FOLLOWER]: Advancing to term 3
07:59:55.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:55.110522  9237 raft_consensus.cc:515] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
07:59:55.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:55.111140  9237 leader_election.cc:290] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [CANDIDATE]: Term 3 election: Requested vote from peers bcfe780000f44bb5ba6ab6347414c393 (127.8.29.129:44331), 21124dcbd9914973a616d9ed69870d0c (127.8.29.130:37019)
07:59:55.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:55.111907  9176 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1d28f54265f409eb6d7b87d41754485" candidate_uuid: "44002126d86d4b42930c65d831be6728" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "21124dcbd9914973a616d9ed69870d0c"
07:59:55.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:55.112175  9176 raft_consensus.cc:3060] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 1 FOLLOWER]: Advancing to term 3
07:59:55.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:55.112882  8881 leader_election.cc:336] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer bcfe780000f44bb5ba6ab6347414c393 (127.8.29.129:44331): Network error: Client connection negotiation failed: client connection to 127.8.29.129:44331: connect: Connection refused (error 111)
07:59:55.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:55.116110  9176 raft_consensus.cc:2468] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44002126d86d4b42930c65d831be6728 in term 3.
07:59:55.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:55.116648  8878 leader_election.cc:304] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 21124dcbd9914973a616d9ed69870d0c, 44002126d86d4b42930c65d831be6728; no voters: bcfe780000f44bb5ba6ab6347414c393
07:59:55.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:55.116966  9237 raft_consensus.cc:2804] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 3 FOLLOWER]: Leader election won for term 3
07:59:55.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:55.117489  9237 raft_consensus.cc:697] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 3 LEADER]: Becoming Leader. State: Replica: 44002126d86d4b42930c65d831be6728, State: Running, Role: LEADER
07:59:55.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:55.118080  9237 consensus_queue.cc:237] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
07:59:55.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:55.122321  8494 catalog_manager.cc:5696] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 reported cstate change: term changed from 2 to 3, leader changed from bcfe780000f44bb5ba6ab6347414c393 (127.8.29.129) to 44002126d86d4b42930c65d831be6728 (127.8.29.131). New cstate: current_term: 3 leader_uuid: "44002126d86d4b42930c65d831be6728" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } health_report { overall_health: HEALTHY } } }
07:59:55.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:55.631829  9176 raft_consensus.cc:1275] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 3 FOLLOWER]: Refusing update from remote peer 44002126d86d4b42930c65d831be6728: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
07:59:55.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:55.632742  9255 consensus_queue.cc:1048] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 4, Time since last communication: 0.000s
07:59:55.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:55.633733  8881 consensus_peers.cc:597] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 -> Peer bcfe780000f44bb5ba6ab6347414c393 (127.8.29.129:44331): Couldn't send request to peer bcfe780000f44bb5ba6ab6347414c393. Status: Network error: Client connection negotiation failed: client connection to 127.8.29.129:44331: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
07:59:56.185 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
07:59:56.186 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019),44002126d86d4b42930c65d831be6728(127.8.29.131:37185)[L],bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)]}]
07:59:56.186 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=14, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=2459), Traces: [0ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [2ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [2ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [18ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [20ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [22ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [22ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [35ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [38ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [40ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [41ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [42ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [75ms] refreshing cache from master, [76ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [79ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [80ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [82ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [82ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [135ms] refreshing cache from master, [135ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [138ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [140ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [142ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [142ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [315ms] refreshing cache from master, [315ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [318ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [320ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [322ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [322ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [935ms] refreshing cache from master, [935ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [938ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [939ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [941ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [941ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [2455ms] refreshing cache from master, [2455ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [2458ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@1296239173(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@c8b3a09 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1262515787 after retry RPC@783402194) -> (continuation of Deferred@317394661 after retry RPC@844943166) -> (continuation of Deferred@970405587 after retry RPC@1831222248) -> (continuation of Deferred@1781049071 after retry RPC@777192796) -> (continuation of Deferred@1261370003 after retry RPC@265080646) -> (continuation of Deferred@2040206130 after retry RPC@281302456), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@17ee5f5e -> wakeup thread Time-limited test -> (continuation of Deferred@1262515787 after retry RPC@783402194) -> (continuation of Deferred@317394661 after retry RPC@844943166) -> (continuation of Deferred@970405587 after retry RPC@1831222248) -> (continuation of Deferred@1781049071 after retry RPC@777192796) -> (continuation of Deferred@1261370003 after retry RPC@265080646) -> (continuation of Deferred@2040206130 after retry RPC@281302456))) after lookup
07:59:56.187 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] connecting to peer
07:59:56.188 [DEBUG - kudu-nio-2] (Connection.java:757) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] Successfully connected to peer
07:59:56.207 [DEBUG - kudu-nio-2] (Negotiator.java:919) Authenticated connection [id: 0x91ea82f5, L:/127.0.0.1:43574 - R:/127.8.29.131:37185] using TOKEN/null
07:59:56.243 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.8.29.129:44331
07:59:56.244 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

07:59:56.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.245340  8310 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskivGXg8/build/asan/bin/kudu
07:59:56.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskivGXg8/build/asan/bin/kudu
07:59:56.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/wal
07:59:56.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/data
07:59:56.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
07:59:56.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
07:59:56.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
07:59:56.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:59:56.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:59:56.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:59:56.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/logs
07:59:56.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/data/info.pb
07:59:56.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:59:56.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
07:59:56.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
07:59:56.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
07:59:56.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
07:59:56.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:59:56.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:59:56.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:59:56.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
07:59:56.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
07:59:56.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.8.29.129:44331
07:59:56.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.8.29.129
07:59:56.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.8.29.190:44257,127.8.29.189:39965,127.8.29.188:37745
07:59:56.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=38159
07:59:56.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.8.29.129
07:59:56.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.8.29.148:45029
07:59:56.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:59:56.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:59:56.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
07:59:56.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:56.594590  9276 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
07:59:56.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:56.594908  9276 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
07:59:56.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:56.594988  9276 flags.cc:432] Enabled unsafe flag: --never_fsync=true
07:59:56.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:56.603477  9276 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
07:59:56.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:56.603739  9276 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.129
07:59:56.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.614527  9276 tablet_server_runner.cc:78] Tablet server non-default flags:
07:59:56.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:59:56.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.8.29.148:45029
07:59:56.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:59:56.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:59:56.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/data
07:59:56.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/wal
07:59:56.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:59:56.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:59:56.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.8.29.129:44331
07:59:56.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
07:59:56.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:59:56.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:59:56.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/data/info.pb
07:59:56.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.8.29.129
07:59:56.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=38159
07:59:56.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.8.29.190:44257,127.8.29.189:39965,127.8.29.188:37745
07:59:56.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
07:59:56.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:59:56.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
07:59:56.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
07:59:56.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:59:56.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.8.29.129
07:59:56.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/logs
07:59:56.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:59:56.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
07:59:56.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
07:59:56.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
07:59:56.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
07:59:56.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
07:59:56.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
07:59:56.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 07:43:24 UTC on bdcb31816ec0
07:59:56.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11954
07:59:56.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
07:59:56.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.616779  9276 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
07:59:56.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.618837  9276 file_cache.cc:492] Constructed file cache file cache with capacity 419430
07:59:56.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:56.629540  9282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:59:56.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:56.629763  9281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:59:56.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:56.629632  9284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:59:56.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.630904  9276 server_base.cc:1061] running on GCE node
07:59:56.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.631623  9276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
07:59:56.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.632706  9276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
07:59:56.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.634029  9276 hybrid_clock.cc:648] HybridClock initialized: now 1779091196633930 us; error 84 us; skew 500 ppm
07:59:56.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.634440  9276 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
07:59:56.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.636853  9276 webserver.cc:533] Webserver started at http://127.8.29.129:38159/ using document root <none> and password file <none>
07:59:56.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.637656  9276 fs_manager.cc:362] Metadata directory not provided
07:59:56.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.637816  9276 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
07:59:56.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.643868  9276 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.000s
07:59:56.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.647200  9290 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
07:59:56.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.649394  9276 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
07:59:56.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.649752  9276 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/data,/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/wal
07:59:56.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "bcfe780000f44bb5ba6ab6347414c393"
07:59:56.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 07:59:45 on dist-test-slave-c1kb"
07:59:56.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.650833  9276 fs_report.cc:389] FS layout report
07:59:56.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
07:59:56.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/wal
07:59:56.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/wal
07:59:56.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/data/data
07:59:56.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
07:59:56.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
07:59:56.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
07:59:56.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
07:59:56.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
07:59:56.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
07:59:56.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
07:59:56.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
07:59:56.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
07:59:56.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
07:59:56.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
07:59:56.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.671507  9276 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
07:59:56.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.672549  9276 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
07:59:56.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.672900  9276 kserver.cc:163] Server-wide thread pool size limit: 3276
07:59:56.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.674084  9276 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
07:59:56.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.676541  9297 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
07:59:56.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.682600  9276 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
07:59:56.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.682842  9276 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.007s	user 0.000s	sys 0.001s
07:59:56.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.683091  9276 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
07:59:56.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.686156  9276 ts_tablet_manager.cc:616] Registered 1 tablets
07:59:56.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.686297  9276 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.000s	sys 0.000s
07:59:56.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.686784  9297 tablet_bootstrap.cc:492] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393: Bootstrap starting.
07:59:56.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.713188  9297 log.cc:826] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393: Log is configured to *not* fsync() on all Append() calls
07:59:56.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.732514  9297 tablet_bootstrap.cc:492] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=6 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
07:59:56.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.733071  9297 tablet_bootstrap.cc:492] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393: Bootstrap complete.
07:59:56.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.734830  9297 ts_tablet_manager.cc:1403] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393: Time spent bootstrapping tablet: real 0.048s	user 0.031s	sys 0.017s
07:59:56.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.740483  9276 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.129:44331
07:59:56.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.740535  9404 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.29.129:44331 every 8 connection(s)
07:59:56.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.742447  9276 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/data/info.pb
07:59:56.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.743826  8310 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskivGXg8/build/asan/bin/kudu as pid 9276
07:59:56.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.742954  9297 raft_consensus.cc:359] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
07:59:56.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.744083  9297 raft_consensus.cc:740] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcfe780000f44bb5ba6ab6347414c393, State: Initialized, Role: FOLLOWER
07:59:56.744 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
07:59:56.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.745153  9297 consensus_queue.cc:260] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
07:59:56.763 [DEBUG - kudu-nio-2] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 9 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328, tablet=e1d28f54265f409eb6d7b87d41754485, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
07:59:56.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.763655  9297 ts_tablet_manager.cc:1434] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393: Time spent starting tablet: real 0.029s	user 0.013s	sys 0.000s
07:59:56.764 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.8.29.131:37185
07:59:56.765 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

07:59:56.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.765327  8310 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskivGXg8/build/asan/bin/kudu pid 8863
07:59:56.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.798380  9405 heartbeater.cc:344] Connected to a master server at 127.8.29.188:37745
07:59:56.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.798893  9405 heartbeater.cc:461] Registering TS with master...
07:59:56.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.800354  9405 heartbeater.cc:507] Master 127.8.29.188:37745 requested a full tablet report, sending...
07:59:56.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.803953  8494 ts_manager.cc:194] Re-registered known tserver with Master: bcfe780000f44bb5ba6ab6347414c393 (127.8.29.129:44331)
07:59:56.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.805519  8494 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.8.29.129:36759
07:59:56.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.811265  9405 heartbeater.cc:499] Master 127.8.29.188:37745 was elected leader, sending a full tablet report...
07:59:56.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.819218  8985 generic_service.cc:196] Checking for leaks (request via RPC)
07:59:56.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.828574  9411 heartbeater.cc:344] Connected to a master server at 127.8.29.189:39965
07:59:56.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.828763  9411 heartbeater.cc:461] Registering TS with master...
07:59:56.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.829106  9411 heartbeater.cc:507] Master 127.8.29.189:39965 requested a full tablet report, sending...
07:59:56.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.829957  9406 heartbeater.cc:344] Connected to a master server at 127.8.29.190:44257
07:59:56.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.830179  9406 heartbeater.cc:461] Registering TS with master...
07:59:56.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.830462  9406 heartbeater.cc:507] Master 127.8.29.190:44257 requested a full tablet report, sending...
07:59:56.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.830605  8424 ts_manager.cc:194] Re-registered known tserver with Master: bcfe780000f44bb5ba6ab6347414c393 (127.8.29.129:44331)
07:59:56.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.831947  8353 ts_manager.cc:194] Re-registered known tserver with Master: bcfe780000f44bb5ba6ab6347414c393 (127.8.29.129:44331)
07:59:56.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:56.931205  8310 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskivGXg8/build/asan/bin/kudu with pid 8863
07:59:56.950 [DEBUG - kudu-nio-2] (Connection.java:239) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] handling channelInactive
07:59:56.950 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] cleaning up while in state READY due to: connection closed
07:59:56.950 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
07:59:56.951 [DEBUG - Time-limited test] (Connection.java:748) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] connecting to peer
07:59:56.952 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
07:59:56.952 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] cleaning up while in state CONNECTING due to: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
07:59:56.953 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 44002126d86d4b42930c65d831be6728(127.8.29.131:37185) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
07:59:56.966 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
07:59:56.966 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019),44002126d86d4b42930c65d831be6728(127.8.29.131:37185)[L],bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)]}]
07:59:56.967 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=2, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=16), Traces: [0ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [2ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [2ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [15ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@2133962722(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@4dea5c29 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@57fb58c7 -> wakeup thread Time-limited test)) after lookup
07:59:56.968 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] connecting to peer
07:59:56.969 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
07:59:56.969 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] cleaning up while in state CONNECTING due to: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
07:59:56.969 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 44002126d86d4b42930c65d831be6728(127.8.29.131:37185) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
07:59:56.986 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
07:59:56.986 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019),44002126d86d4b42930c65d831be6728(127.8.29.131:37185)[L],bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)]}]
07:59:56.987 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=4, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=36), Traces: [0ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [2ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [2ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [15ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [16ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [18ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [18ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [34ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@2133962722(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@4dea5c29 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1973592499 after retry RPC@2014507680), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@57fb58c7 -> wakeup thread Time-limited test -> (continuation of Deferred@1973592499 after retry RPC@2014507680))) after lookup
07:59:56.987 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] connecting to peer
07:59:56.989 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
07:59:56.989 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] cleaning up while in state CONNECTING due to: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
07:59:56.989 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 44002126d86d4b42930c65d831be6728(127.8.29.131:37185) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
07:59:57.025 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
07:59:57.026 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019),44002126d86d4b42930c65d831be6728(127.8.29.131:37185)[L],bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)]}]
07:59:57.026 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=6, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=75), Traces: [0ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [2ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [2ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [15ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [16ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [18ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [18ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [34ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [36ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [38ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [38ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [71ms] refreshing cache from master, [71ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [74ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@2133962722(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@4dea5c29 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1973592499 after retry RPC@2014507680) -> (continuation of Deferred@233229336 after retry RPC@1645127926), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@57fb58c7 -> wakeup thread Time-limited test -> (continuation of Deferred@1973592499 after retry RPC@2014507680) -> (continuation of Deferred@233229336 after retry RPC@1645127926))) after lookup
07:59:57.027 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] connecting to peer
07:59:57.028 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
07:59:57.028 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] cleaning up while in state CONNECTING due to: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
07:59:57.028 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 44002126d86d4b42930c65d831be6728(127.8.29.131:37185) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
07:59:57.126 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
07:59:57.126 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019),44002126d86d4b42930c65d831be6728(127.8.29.131:37185)[L],bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)]}]
07:59:57.126 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=8, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=175), Traces: [0ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [2ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [2ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [15ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [16ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [18ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [18ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [34ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [36ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [38ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [38ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [71ms] refreshing cache from master, [71ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [74ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [76ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [77ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [77ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [171ms] refreshing cache from master, [171ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [174ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@2133962722(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@4dea5c29 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1973592499 after retry RPC@2014507680) -> (continuation of Deferred@233229336 after retry RPC@1645127926) -> (continuation of Deferred@944663528 after retry RPC@797977871), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@57fb58c7 -> wakeup thread Time-limited test -> (continuation of Deferred@1973592499 after retry RPC@2014507680) -> (continuation of Deferred@233229336 after retry RPC@1645127926) -> (continuation of Deferred@944663528 after retry RPC@797977871))) after lookup
07:59:57.127 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] connecting to peer
07:59:57.128 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
07:59:57.128 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] cleaning up while in state CONNECTING due to: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
07:59:57.128 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 44002126d86d4b42930c65d831be6728(127.8.29.131:37185) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
07:59:57.286 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
07:59:57.286 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019),44002126d86d4b42930c65d831be6728(127.8.29.131:37185)[L],bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)]}]
07:59:57.286 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=10, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=335), Traces: [0ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [2ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [2ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [15ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [16ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [18ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [18ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [34ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [36ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [38ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [38ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [71ms] refreshing cache from master, [71ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [74ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [76ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [77ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [77ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [171ms] refreshing cache from master, [171ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [174ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [176ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [177ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [177ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [331ms] refreshing cache from master, [331ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [334ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@2133962722(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@4dea5c29 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1973592499 after retry RPC@2014507680) -> (continuation of Deferred@233229336 after retry RPC@1645127926) -> (continuation of Deferred@944663528 after retry RPC@797977871) -> (continuation of Deferred@2084509577 after retry RPC@1382655316), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@57fb58c7 -> wakeup thread Time-limited test -> (continuation of Deferred@1973592499 after retry RPC@2014507680) -> (continuation of Deferred@233229336 after retry RPC@1645127926) -> (continuation of Deferred@944663528 after retry RPC@797977871) -> (continuation of Deferred@2084509577 after retry RPC@1382655316))) after lookup
07:59:57.287 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] connecting to peer
07:59:57.288 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
07:59:57.288 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] cleaning up while in state CONNECTING due to: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
07:59:57.288 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 44002126d86d4b42930c65d831be6728(127.8.29.131:37185) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
07:59:58.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:58.263502  9421 raft_consensus.cc:493] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
07:59:58.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:58.263837  9421 raft_consensus.cc:515] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
07:59:58.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:58.265694  9421 leader_election.cc:290] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 21124dcbd9914973a616d9ed69870d0c (127.8.29.130:37019), 44002126d86d4b42930c65d831be6728 (127.8.29.131:37185)
07:59:58.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:58.267372  9292 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.29.131:37185: connect: Connection refused (error 111)
07:59:58.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:58.269248  9292 leader_election.cc:336] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 44002126d86d4b42930c65d831be6728 (127.8.29.131:37185): Network error: Client connection negotiation failed: client connection to 127.8.29.131:37185: connect: Connection refused (error 111)
07:59:58.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:58.273106  9176 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1d28f54265f409eb6d7b87d41754485" candidate_uuid: "bcfe780000f44bb5ba6ab6347414c393" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "21124dcbd9914973a616d9ed69870d0c" is_pre_election: true
07:59:58.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:58.274070  9291 leader_election.cc:304] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcfe780000f44bb5ba6ab6347414c393; no voters: 21124dcbd9914973a616d9ed69870d0c, 44002126d86d4b42930c65d831be6728
07:59:58.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:58.274526  9421 raft_consensus.cc:2749] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
07:59:58.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:58.310467  9426 raft_consensus.cc:493] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 44002126d86d4b42930c65d831be6728)
07:59:58.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:58.310758  9426 raft_consensus.cc:515] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
07:59:58.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:58.312460  9426 leader_election.cc:290] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcfe780000f44bb5ba6ab6347414c393 (127.8.29.129:44331), 44002126d86d4b42930c65d831be6728 (127.8.29.131:37185)
07:59:58.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:58.313777  9109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.29.131:37185: connect: Connection refused (error 111)
07:59:58.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:58.315522  9109 leader_election.cc:336] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 44002126d86d4b42930c65d831be6728 (127.8.29.131:37185): Network error: Client connection negotiation failed: client connection to 127.8.29.131:37185: connect: Connection refused (error 111)
07:59:58.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:58.322359  9359 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1d28f54265f409eb6d7b87d41754485" candidate_uuid: "21124dcbd9914973a616d9ed69870d0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "bcfe780000f44bb5ba6ab6347414c393" is_pre_election: true
07:59:58.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:58.322824  9359 raft_consensus.cc:2468] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21124dcbd9914973a616d9ed69870d0c in term 2.
07:59:58.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:58.324111  9111 leader_election.cc:304] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 21124dcbd9914973a616d9ed69870d0c, bcfe780000f44bb5ba6ab6347414c393; no voters: 44002126d86d4b42930c65d831be6728
07:59:58.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:58.324601  9426 raft_consensus.cc:2804] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 3 FOLLOWER]: Leader pre-election won for term 4
07:59:58.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:58.324748  9426 raft_consensus.cc:493] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 3 FOLLOWER]: Starting leader election (detected failure of leader 44002126d86d4b42930c65d831be6728)
07:59:58.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:58.324862  9426 raft_consensus.cc:3060] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 3 FOLLOWER]: Advancing to term 4
07:59:58.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:58.327066  9426 raft_consensus.cc:515] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
07:59:58.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:58.327805  9426 leader_election.cc:290] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [CANDIDATE]: Term 4 election: Requested vote from peers bcfe780000f44bb5ba6ab6347414c393 (127.8.29.129:44331), 44002126d86d4b42930c65d831be6728 (127.8.29.131:37185)
07:59:58.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:58.328406  9359 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1d28f54265f409eb6d7b87d41754485" candidate_uuid: "21124dcbd9914973a616d9ed69870d0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "bcfe780000f44bb5ba6ab6347414c393"
07:59:58.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:58.328622  9359 raft_consensus.cc:3060] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 2 FOLLOWER]: Advancing to term 4
07:59:58.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:58.329371  9109 leader_election.cc:336] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 44002126d86d4b42930c65d831be6728 (127.8.29.131:37185): Network error: Client connection negotiation failed: client connection to 127.8.29.131:37185: connect: Connection refused (error 111)
07:59:58.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:58.332383  9359 raft_consensus.cc:2468] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21124dcbd9914973a616d9ed69870d0c in term 4.
07:59:58.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:58.332839  9111 leader_election.cc:304] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 21124dcbd9914973a616d9ed69870d0c, bcfe780000f44bb5ba6ab6347414c393; no voters: 44002126d86d4b42930c65d831be6728
07:59:58.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:58.333179  9426 raft_consensus.cc:2804] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 4 FOLLOWER]: Leader election won for term 4
07:59:58.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:58.333643  9426 raft_consensus.cc:697] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 4 LEADER]: Becoming Leader. State: Replica: 21124dcbd9914973a616d9ed69870d0c, State: Running, Role: LEADER
07:59:58.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:58.334175  9426 consensus_queue.cc:237] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
07:59:58.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:58.338016  8494 catalog_manager.cc:5696] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c reported cstate change: term changed from 3 to 4, leader changed from 44002126d86d4b42930c65d831be6728 (127.8.29.131) to 21124dcbd9914973a616d9ed69870d0c (127.8.29.130). New cstate: current_term: 4 leader_uuid: "21124dcbd9914973a616d9ed69870d0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } health_report { overall_health: UNKNOWN } } }
07:59:58.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:58.850118  9359 raft_consensus.cc:1275] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 4 FOLLOWER]: Refusing update from remote peer 21124dcbd9914973a616d9ed69870d0c: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 4 index: 13. (index mismatch)
07:59:58.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:58.851159  9443 consensus_queue.cc:1048] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 8, Time since last communication: 0.000s
07:59:58.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:58.873967  9109 consensus_peers.cc:597] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c -> Peer 44002126d86d4b42930c65d831be6728 (127.8.29.131:37185): Couldn't send request to peer 44002126d86d4b42930c65d831be6728. Status: Network error: Client connection negotiation failed: client connection to 127.8.29.131:37185: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
07:59:59.206 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
07:59:59.206 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)[L],44002126d86d4b42930c65d831be6728(127.8.29.131:37185),bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)]}]
07:59:59.206 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=12, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=2255), Traces: [0ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [2ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [2ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [15ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [16ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [18ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [18ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [34ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [36ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [38ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [38ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [71ms] refreshing cache from master, [71ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [74ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [76ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [77ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [77ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [171ms] refreshing cache from master, [171ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [174ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [176ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [177ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [177ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [331ms] refreshing cache from master, [331ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [334ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [336ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [337ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [337ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [2251ms] refreshing cache from master, [2251ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [2254ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@2133962722(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@4dea5c29 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1973592499 after retry RPC@2014507680) -> (continuation of Deferred@233229336 after retry RPC@1645127926) -> (continuation of Deferred@944663528 after retry RPC@797977871) -> (continuation of Deferred@2084509577 after retry RPC@1382655316) -> (continuation of Deferred@546695301 after retry RPC@1649494066), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@57fb58c7 -> wakeup thread Time-limited test -> (continuation of Deferred@1973592499 after retry RPC@2014507680) -> (continuation of Deferred@233229336 after retry RPC@1645127926) -> (continuation of Deferred@944663528 after retry RPC@797977871) -> (continuation of Deferred@2084509577 after retry RPC@1382655316) -> (continuation of Deferred@546695301 after retry RPC@1649494066))) after lookup
07:59:59.207 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] connecting to peer
07:59:59.208 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] Successfully connected to peer
07:59:59.230 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x1965f020, L:/127.0.0.1:46638 - R:/127.8.29.130:37019] using TOKEN/null
07:59:59.257 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.8.29.131:37185
07:59:59.258 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

07:59:59.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.258802  8310 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskivGXg8/build/asan/bin/kudu
07:59:59.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskivGXg8/build/asan/bin/kudu
07:59:59.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/wal
07:59:59.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/data
07:59:59.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
07:59:59.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
07:59:59.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
07:59:59.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:59:59.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:59:59.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:59:59.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/logs
07:59:59.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/data/info.pb
07:59:59.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:59:59.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
07:59:59.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
07:59:59.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
07:59:59.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
07:59:59.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:59:59.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:59:59.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:59:59.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
07:59:59.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
07:59:59.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.8.29.131:37185
07:59:59.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.8.29.131
07:59:59.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.8.29.190:44257,127.8.29.189:39965,127.8.29.188:37745
07:59:59.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42487
07:59:59.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.8.29.131
07:59:59.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.8.29.148:45029
07:59:59.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:59:59.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:59:59.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
07:59:59.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:59.616786  9457 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
07:59:59.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:59.617189  9457 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
07:59:59.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:59.617278  9457 flags.cc:432] Enabled unsafe flag: --never_fsync=true
07:59:59.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:59.626461  9457 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
07:59:59.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:59.626645  9457 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.131
07:59:59.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.637842  9457 tablet_server_runner.cc:78] Tablet server non-default flags:
07:59:59.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
07:59:59.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.8.29.148:45029
07:59:59.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
07:59:59.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
07:59:59.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/data
07:59:59.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/wal
07:59:59.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
07:59:59.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
07:59:59.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.8.29.131:37185
07:59:59.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
07:59:59.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
07:59:59.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
07:59:59.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/data/info.pb
07:59:59.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.8.29.131
07:59:59.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42487
07:59:59.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.8.29.190:44257,127.8.29.189:39965,127.8.29.188:37745
07:59:59.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
07:59:59.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
07:59:59.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
07:59:59.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
07:59:59.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
07:59:59.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.8.29.131
07:59:59.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/logs
07:59:59.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
07:59:59.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
07:59:59.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
07:59:59.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
07:59:59.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
07:59:59.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
07:59:59.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
07:59:59.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 07:43:24 UTC on bdcb31816ec0
07:59:59.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11954
07:59:59.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
07:59:59.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.639658  9457 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
07:59:59.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.641539  9457 file_cache.cc:492] Constructed file cache file cache with capacity 419430
07:59:59.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:59.652179  9463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:59:59.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:59.652362  9465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:59:59.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 07:59:59.652463  9462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
07:59:59.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.653515  9457 server_base.cc:1061] running on GCE node
07:59:59.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.654163  9457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
07:59:59.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.655683  9457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
07:59:59.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.656944  9457 hybrid_clock.cc:648] HybridClock initialized: now 1779091199656879 us; error 59 us; skew 500 ppm
07:59:59.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.657245  9457 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
07:59:59.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.659862  9457 webserver.cc:533] Webserver started at http://127.8.29.131:42487/ using document root <none> and password file <none>
07:59:59.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.660633  9457 fs_manager.cc:362] Metadata directory not provided
07:59:59.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.660739  9457 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
07:59:59.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.666639  9457 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.000s
07:59:59.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.669636  9471 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
07:59:59.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.671205  9457 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
07:59:59.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.671344  9457 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/data,/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/wal
07:59:59.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "44002126d86d4b42930c65d831be6728"
07:59:59.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 07:59:46 on dist-test-slave-c1kb"
07:59:59.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.672079  9457 fs_report.cc:389] FS layout report
07:59:59.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
07:59:59.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/wal
07:59:59.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/wal
07:59:59.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/data/data
07:59:59.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
07:59:59.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
07:59:59.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
07:59:59.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
07:59:59.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
07:59:59.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
07:59:59.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
07:59:59.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
07:59:59.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
07:59:59.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
07:59:59.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
07:59:59.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.713920  9457 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
07:59:59.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.715255  9457 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
07:59:59.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.715648  9457 kserver.cc:163] Server-wide thread pool size limit: 3276
07:59:59.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.716964  9457 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
07:59:59.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.720175  9478 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
07:59:59.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.726778  9457 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
07:59:59.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.726934  9457 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.007s	user 0.001s	sys 0.000s
07:59:59.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.727108  9457 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
07:59:59.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.732465  9457 ts_tablet_manager.cc:616] Registered 1 tablets
07:59:59.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.732610  9457 ts_tablet_manager.cc:595] Time spent register tablets: real 0.006s	user 0.005s	sys 0.000s
07:59:59.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.733443  9478 tablet_bootstrap.cc:492] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728: Bootstrap starting.
07:59:59.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.763942  9478 log.cc:826] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728: Log is configured to *not* fsync() on all Append() calls
07:59:59.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.788043  9478 tablet_bootstrap.cc:492] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=9 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
07:59:59.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.789358  9478 tablet_bootstrap.cc:492] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728: Bootstrap complete.
07:59:59.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.791558  9478 ts_tablet_manager.cc:1403] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728: Time spent bootstrapping tablet: real 0.059s	user 0.046s	sys 0.006s
07:59:59.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.799993  9478 raft_consensus.cc:359] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
07:59:59.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.800807  9478 raft_consensus.cc:740] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44002126d86d4b42930c65d831be6728, State: Initialized, Role: FOLLOWER
07:59:59.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.801983  9478 consensus_queue.cc:260] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
07:59:59.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.805274  9478 ts_tablet_manager.cc:1434] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728: Time spent starting tablet: real 0.013s	user 0.012s	sys 0.000s
07:59:59.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.809338  9457 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.131:37185
07:59:59.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.809348  9586 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.29.131:37185 every 8 connection(s)
07:59:59.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.811574  9457 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/data/info.pb
07:59:59.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.814034  8310 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskivGXg8/build/asan/bin/kudu as pid 9457
07:59:59.814 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
07:59:59.874 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 12 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328, tablet=e1d28f54265f409eb6d7b87d41754485, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
07:59:59.883 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.8.29.130:37019
07:59:59.884 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

07:59:59.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.884603  8310 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskivGXg8/build/asan/bin/kudu pid 9093
07:59:59.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.889998  9587 heartbeater.cc:344] Connected to a master server at 127.8.29.188:37745
07:59:59.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.890398  9587 heartbeater.cc:461] Registering TS with master...
07:59:59.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.891646  9587 heartbeater.cc:507] Master 127.8.29.188:37745 requested a full tablet report, sending...
07:59:59.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.895807  8494 ts_manager.cc:194] Re-registered known tserver with Master: 44002126d86d4b42930c65d831be6728 (127.8.29.131:37185)
07:59:59.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.897612  8494 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.8.29.131:60651
07:59:59.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.902148  9589 heartbeater.cc:344] Connected to a master server at 127.8.29.189:39965
07:59:59.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.902383  9589 heartbeater.cc:461] Registering TS with master...
07:59:59.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.902930  9589 heartbeater.cc:507] Master 127.8.29.189:39965 requested a full tablet report, sending...
07:59:59.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.905067  9588 heartbeater.cc:344] Connected to a master server at 127.8.29.190:44257
07:59:59.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.905238  9588 heartbeater.cc:461] Registering TS with master...
07:59:59.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.905275  8424 ts_manager.cc:194] Re-registered known tserver with Master: 44002126d86d4b42930c65d831be6728 (127.8.29.131:37185)
07:59:59.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.905566  9588 heartbeater.cc:507] Master 127.8.29.190:44257 requested a full tablet report, sending...
07:59:59.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.907814  8353 ts_manager.cc:194] Re-registered known tserver with Master: 44002126d86d4b42930c65d831be6728 (127.8.29.131:37185)
07:59:59.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 07:59:59.916376  9217 generic_service.cc:196] Checking for leaks (request via RPC)
08:00:00.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:00.028661  8310 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskivGXg8/build/asan/bin/kudu with pid 9093
08:00:00.048 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] handling channelInactive
08:00:00.049 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] cleaning up while in state READY due to: connection closed
08:00:00.049 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:00:00.050 [DEBUG - Time-limited test] (Connection.java:748) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] connecting to peer
08:00:00.051 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:00.051 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] cleaning up while in state CONNECTING due to: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:00.051 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:00.065 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
08:00:00.065 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)[L],44002126d86d4b42930c65d831be6728(127.8.29.131:37185),bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)]}]
08:00:00.065 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=2, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=16), Traces: [0ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [2ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [2ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [15ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@281963779(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@78b6a7f0 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@257872d8 -> wakeup thread Time-limited test)) after lookup
08:00:00.066 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] connecting to peer
08:00:00.067 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:00.067 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] cleaning up while in state CONNECTING due to: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:00.067 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:00.086 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
08:00:00.086 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)[L],44002126d86d4b42930c65d831be6728(127.8.29.131:37185),bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)]}]
08:00:00.086 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=4, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=37), Traces: [0ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [2ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [2ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [15ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [17ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [18ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [18ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [33ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [36ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@281963779(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@78b6a7f0 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@394835285 after retry RPC@1904274382), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@257872d8 -> wakeup thread Time-limited test -> (continuation of Deferred@394835285 after retry RPC@1904274382))) after lookup
08:00:00.087 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] connecting to peer
08:00:00.088 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:00.088 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] cleaning up while in state CONNECTING due to: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:00.088 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:00.125 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
08:00:00.126 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)[L],44002126d86d4b42930c65d831be6728(127.8.29.131:37185),bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)]}]
08:00:00.126 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=6, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=77), Traces: [0ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [2ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [2ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [15ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [17ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [18ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [18ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [33ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [36ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [37ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [39ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [39ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [73ms] refreshing cache from master, [74ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [76ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@281963779(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@78b6a7f0 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@394835285 after retry RPC@1904274382) -> (continuation of Deferred@1851410131 after retry RPC@410132045), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@257872d8 -> wakeup thread Time-limited test -> (continuation of Deferred@394835285 after retry RPC@1904274382) -> (continuation of Deferred@1851410131 after retry RPC@410132045))) after lookup
08:00:00.127 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] connecting to peer
08:00:00.128 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:00.128 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] cleaning up while in state CONNECTING due to: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:00.128 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:00.246 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
08:00:00.246 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)[L],44002126d86d4b42930c65d831be6728(127.8.29.131:37185),bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)]}]
08:00:00.246 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=8, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=197), Traces: [0ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [2ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [2ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [15ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [17ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [18ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [18ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [33ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [36ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [37ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [39ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [39ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [73ms] refreshing cache from master, [74ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [76ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [77ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [79ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [79ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [193ms] refreshing cache from master, [193ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [196ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@281963779(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@78b6a7f0 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@394835285 after retry RPC@1904274382) -> (continuation of Deferred@1851410131 after retry RPC@410132045) -> (continuation of Deferred@519964194 after retry RPC@385283014), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@257872d8 -> wakeup thread Time-limited test -> (continuation of Deferred@394835285 after retry RPC@1904274382) -> (continuation of Deferred@1851410131 after retry RPC@410132045) -> (continuation of Deferred@519964194 after retry RPC@385283014))) after lookup
08:00:00.248 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] connecting to peer
08:00:00.249 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:00.249 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] cleaning up while in state CONNECTING due to: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:00.249 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:00.385 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
08:00:00.385 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)[L],44002126d86d4b42930c65d831be6728(127.8.29.131:37185),bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)]}]
08:00:00.386 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=10, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=336), Traces: [0ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [2ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [2ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [15ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [17ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [18ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [18ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [33ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [36ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [37ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [39ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [39ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [73ms] refreshing cache from master, [74ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [76ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [77ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [79ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [79ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [193ms] refreshing cache from master, [193ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [196ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [198ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [200ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [200ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [332ms] refreshing cache from master, [333ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [335ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@281963779(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@78b6a7f0 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@394835285 after retry RPC@1904274382) -> (continuation of Deferred@1851410131 after retry RPC@410132045) -> (continuation of Deferred@519964194 after retry RPC@385283014) -> (continuation of Deferred@969583397 after retry RPC@411240217), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@257872d8 -> wakeup thread Time-limited test -> (continuation of Deferred@394835285 after retry RPC@1904274382) -> (continuation of Deferred@1851410131 after retry RPC@410132045) -> (continuation of Deferred@519964194 after retry RPC@385283014) -> (continuation of Deferred@969583397 after retry RPC@411240217))) after lookup
08:00:00.387 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] connecting to peer
08:00:00.388 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:00.388 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] cleaning up while in state CONNECTING due to: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:00.388 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:00.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:00.902243  9587 heartbeater.cc:499] Master 127.8.29.188:37745 was elected leader, sending a full tablet report...
08:00:00.926 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
08:00:00.926 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)[L],44002126d86d4b42930c65d831be6728(127.8.29.131:37185),bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)]}]
08:00:00.927 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=12, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=877), Traces: [0ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [2ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [2ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [15ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [17ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [18ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [18ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [33ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [36ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [37ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [39ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [39ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [73ms] refreshing cache from master, [74ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [76ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [77ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [79ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [79ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [193ms] refreshing cache from master, [193ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [196ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [198ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [200ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [200ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [332ms] refreshing cache from master, [333ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [335ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [337ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [339ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [340ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [873ms] refreshing cache from master, [873ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [877ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@281963779(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@78b6a7f0 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@394835285 after retry RPC@1904274382) -> (continuation of Deferred@1851410131 after retry RPC@410132045) -> (continuation of Deferred@519964194 after retry RPC@385283014) -> (continuation of Deferred@969583397 after retry RPC@411240217) -> (continuation of Deferred@1902873150 after retry RPC@1909514437), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@257872d8 -> wakeup thread Time-limited test -> (continuation of Deferred@394835285 after retry RPC@1904274382) -> (continuation of Deferred@1851410131 after retry RPC@410132045) -> (continuation of Deferred@519964194 after retry RPC@385283014) -> (continuation of Deferred@969583397 after retry RPC@411240217) -> (continuation of Deferred@1902873150 after retry RPC@1909514437))) after lookup
08:00:00.928 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] connecting to peer
08:00:00.929 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:00.929 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] cleaning up while in state CONNECTING due to: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:00.929 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:01.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:01.260497  9602 raft_consensus.cc:493] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 21124dcbd9914973a616d9ed69870d0c)
08:00:01.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:01.260690  9602 raft_consensus.cc:515] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
08:00:01.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:01.261399  9602 leader_election.cc:290] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 21124dcbd9914973a616d9ed69870d0c (127.8.29.130:37019), 44002126d86d4b42930c65d831be6728 (127.8.29.131:37185)
08:00:01.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:01.268915  9291 leader_election.cc:336] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 21124dcbd9914973a616d9ed69870d0c (127.8.29.130:37019): Network error: Client connection negotiation failed: client connection to 127.8.29.130:37019: connect: Connection refused (error 111)
08:00:01.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:01.271255  9541 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1d28f54265f409eb6d7b87d41754485" candidate_uuid: "bcfe780000f44bb5ba6ab6347414c393" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "44002126d86d4b42930c65d831be6728" is_pre_election: true
08:00:01.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:01.271828  9541 raft_consensus.cc:2468] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcfe780000f44bb5ba6ab6347414c393 in term 3.
08:00:01.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:01.272893  9292 leader_election.cc:304] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44002126d86d4b42930c65d831be6728, bcfe780000f44bb5ba6ab6347414c393; no voters: 21124dcbd9914973a616d9ed69870d0c
08:00:01.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:01.273313  9602 raft_consensus.cc:2804] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 4 FOLLOWER]: Leader pre-election won for term 5
08:00:01.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:01.273465  9602 raft_consensus.cc:493] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 21124dcbd9914973a616d9ed69870d0c)
08:00:01.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:01.273548  9602 raft_consensus.cc:3060] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 4 FOLLOWER]: Advancing to term 5
08:00:01.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:01.275736  9602 raft_consensus.cc:515] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
08:00:01.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:01.276683  9602 leader_election.cc:290] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [CANDIDATE]: Term 5 election: Requested vote from peers 21124dcbd9914973a616d9ed69870d0c (127.8.29.130:37019), 44002126d86d4b42930c65d831be6728 (127.8.29.131:37185)
08:00:01.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:01.277274  9541 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1d28f54265f409eb6d7b87d41754485" candidate_uuid: "bcfe780000f44bb5ba6ab6347414c393" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "44002126d86d4b42930c65d831be6728"
08:00:01.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:01.277500  9541 raft_consensus.cc:3060] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 3 FOLLOWER]: Advancing to term 5
08:00:01.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:01.277915  9291 leader_election.cc:336] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 21124dcbd9914973a616d9ed69870d0c (127.8.29.130:37019): Network error: Client connection negotiation failed: client connection to 127.8.29.130:37019: connect: Connection refused (error 111)
08:00:01.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:01.282021  9541 raft_consensus.cc:2468] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcfe780000f44bb5ba6ab6347414c393 in term 5.
08:00:01.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:01.282881  9292 leader_election.cc:304] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44002126d86d4b42930c65d831be6728, bcfe780000f44bb5ba6ab6347414c393; no voters: 21124dcbd9914973a616d9ed69870d0c
08:00:01.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:01.283301  9602 raft_consensus.cc:2804] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 5 FOLLOWER]: Leader election won for term 5
08:00:01.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:01.283860  9602 raft_consensus.cc:697] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 5 LEADER]: Becoming Leader. State: Replica: bcfe780000f44bb5ba6ab6347414c393, State: Running, Role: LEADER
08:00:01.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:01.284582  9602 consensus_queue.cc:237] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 4.16, Last appended by leader: 16, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
08:00:01.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:01.288452  8494 catalog_manager.cc:5696] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 reported cstate change: term changed from 4 to 5, leader changed from 21124dcbd9914973a616d9ed69870d0c (127.8.29.130) to bcfe780000f44bb5ba6ab6347414c393 (127.8.29.129). New cstate: current_term: 5 leader_uuid: "bcfe780000f44bb5ba6ab6347414c393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } health_report { overall_health: UNKNOWN } } }
08:00:01.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:01.689890  9291 consensus_peers.cc:597] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 -> Peer 21124dcbd9914973a616d9ed69870d0c (127.8.29.130:37019): Couldn't send request to peer 21124dcbd9914973a616d9ed69870d0c. Status: Network error: Client connection negotiation failed: client connection to 127.8.29.130:37019: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
08:00:01.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:01.902745  9541 raft_consensus.cc:1275] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 5 FOLLOWER]: Refusing update from remote peer bcfe780000f44bb5ba6ab6347414c393: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 5 index: 17. (index mismatch)
08:00:01.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:01.904038  9602 consensus_queue.cc:1048] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 12, Time since last communication: 0.000s
08:00:03.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:03.555171  9291 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.29.130:37019: connect: Connection refused (error 111) [suppressed 6 similar messages]
08:00:04.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:04.016820  9291 consensus_peers.cc:597] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 -> Peer 21124dcbd9914973a616d9ed69870d0c (127.8.29.130:37019): Couldn't send request to peer 21124dcbd9914973a616d9ed69870d0c. Status: Network error: Client connection negotiation failed: client connection to 127.8.29.130:37019: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
08:00:04.766 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
08:00:04.766 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019),44002126d86d4b42930c65d831be6728(127.8.29.131:37185),bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)[L]]}]
08:00:04.767 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=14, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=4717), Traces: [0ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [2ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [2ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [15ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [17ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [18ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [18ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [33ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [36ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [37ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [39ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [39ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [73ms] refreshing cache from master, [74ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [76ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [77ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [79ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [79ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [193ms] refreshing cache from master, [193ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [196ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [198ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [200ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [200ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [332ms] refreshing cache from master, [333ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [335ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [337ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [339ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [340ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [873ms] refreshing cache from master, [873ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [877ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [878ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [880ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [880ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [4713ms] refreshing cache from master, [4713ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [4717ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@281963779(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@78b6a7f0 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@394835285 after retry RPC@1904274382) -> (continuation of Deferred@1851410131 after retry RPC@410132045) -> (continuation of Deferred@519964194 after retry RPC@385283014) -> (continuation of Deferred@969583397 after retry RPC@411240217) -> (continuation of Deferred@1902873150 after retry RPC@1909514437) -> (continuation of Deferred@1008431048 after retry RPC@987963587), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@257872d8 -> wakeup thread Time-limited test -> (continuation of Deferred@394835285 after retry RPC@1904274382) -> (continuation of Deferred@1851410131 after retry RPC@410132045) -> (continuation of Deferred@519964194 after retry RPC@385283014) -> (continuation of Deferred@969583397 after retry RPC@411240217) -> (continuation of Deferred@1902873150 after retry RPC@1909514437) -> (continuation of Deferred@1008431048 after retry RPC@987963587))) after lookup
08:00:04.767 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] connecting to peer
08:00:04.768 [DEBUG - kudu-nio-1] (Connection.java:757) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] Successfully connected to peer
08:00:04.799 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0xe0862c6b, L:/127.0.0.1:48634 - R:/127.8.29.129:44331] using TOKEN/null
08:00:04.843 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.8.29.130:37019
08:00:04.843 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

08:00:04.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:04.844503  8310 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskivGXg8/build/asan/bin/kudu
08:00:04.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskivGXg8/build/asan/bin/kudu
08:00:04.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/wal
08:00:04.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/data
08:00:04.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:00:04.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:00:04.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:00:04.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:00:04.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:00:04.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:00:04.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/logs
08:00:04.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/data/info.pb
08:00:04.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:00:04.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:00:04.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:00:04.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:00:04.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:00:04.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:00:04.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:00:04.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:00:04.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:00:04.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:00:04.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.8.29.130:37019
08:00:04.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.8.29.130
08:00:04.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.8.29.190:44257,127.8.29.189:39965,127.8.29.188:37745
08:00:04.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44565
08:00:04.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.8.29.130
08:00:04.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.8.29.148:45029
08:00:04.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:00:04.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:00:04.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:00:05.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:05.241135  9644 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:00:05.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:05.241490  9644 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:00:05.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:05.241567  9644 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:00:05.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:05.251325  9644 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:00:05.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:05.251586  9644 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.130
08:00:05.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.265106  9644 tablet_server_runner.cc:78] Tablet server non-default flags:
08:00:05.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:00:05.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.8.29.148:45029
08:00:05.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:00:05.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:00:05.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/data
08:00:05.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/wal
08:00:05.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:00:05.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:00:05.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.8.29.130:37019
08:00:05.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:00:05.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:00:05.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:00:05.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/data/info.pb
08:00:05.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.8.29.130
08:00:05.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44565
08:00:05.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.8.29.190:44257,127.8.29.189:39965,127.8.29.188:37745
08:00:05.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:00:05.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:00:05.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:00:05.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:00:05.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:00:05.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.8.29.130
08:00:05.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/logs
08:00:05.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:00:05.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:00:05.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:00:05.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:00:05.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:00:05.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
08:00:05.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:00:05.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 07:43:24 UTC on bdcb31816ec0
08:00:05.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11954
08:00:05.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
08:00:05.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.267058  9644 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:00:05.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.269943  9644 file_cache.cc:492] Constructed file cache file cache with capacity 419430
08:00:05.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:05.283547  9652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:00:05.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:05.283571  9651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:00:05.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:05.283533  9654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:00:05.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.284065  9644 server_base.cc:1061] running on GCE node
08:00:05.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.285033  9644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:00:05.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.286321  9644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:00:05.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.287765  9644 hybrid_clock.cc:648] HybridClock initialized: now 1779091205287680 us; error 69 us; skew 500 ppm
08:00:05.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.288265  9644 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:00:05.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.290868  9644 webserver.cc:533] Webserver started at http://127.8.29.130:44565/ using document root <none> and password file <none>
08:00:05.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.291863  9644 fs_manager.cc:362] Metadata directory not provided
08:00:05.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.292013  9644 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:00:05.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.298998  9644 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.006s	sys 0.000s
08:00:05.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.302862  9660 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:00:05.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.305150  9644 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
08:00:05.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.305392  9644 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/data,/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/wal
08:00:05.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "21124dcbd9914973a616d9ed69870d0c"
08:00:05.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 07:59:46 on dist-test-slave-c1kb"
08:00:05.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.306644  9644 fs_report.cc:389] FS layout report
08:00:05.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:00:05.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/wal
08:00:05.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/wal
08:00:05.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/data/data
08:00:05.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:00:05.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:00:05.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:00:05.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:00:05.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:00:05.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:00:05.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:00:05.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:00:05.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:00:05.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:00:05.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:00:05.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.369387  9644 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:00:05.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.370453  9644 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:00:05.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.370728  9644 kserver.cc:163] Server-wide thread pool size limit: 3276
08:00:05.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.371788  9644 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
08:00:05.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.374179  9667 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
08:00:05.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.381273  9644 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
08:00:05.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.381398  9644 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.008s	user 0.001s	sys 0.000s
08:00:05.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.381517  9644 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
08:00:05.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.385154  9644 ts_tablet_manager.cc:616] Registered 1 tablets
08:00:05.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.385267  9644 ts_tablet_manager.cc:595] Time spent register tablets: real 0.004s	user 0.002s	sys 0.000s
08:00:05.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.385957  9667 tablet_bootstrap.cc:492] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c: Bootstrap starting.
08:00:05.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.413514  9667 log.cc:826] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c: Log is configured to *not* fsync() on all Append() calls
08:00:05.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.433805  9644 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.130:37019
08:00:05.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.433861  9774 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.29.130:37019 every 8 connection(s)
08:00:05.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.435458  9644 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/data/info.pb
08:00:05.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.439551  8310 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskivGXg8/build/asan/bin/kudu as pid 9644
08:00:05.440 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:00:05.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.446345  9667 tablet_bootstrap.cc:492] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=12 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
08:00:05.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.447233  9667 tablet_bootstrap.cc:492] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c: Bootstrap complete.
08:00:05.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.449800  9667 ts_tablet_manager.cc:1403] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c: Time spent bootstrapping tablet: real 0.064s	user 0.042s	sys 0.017s
08:00:05.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.460599  9667 raft_consensus.cc:359] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
08:00:05.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.462384  9667 raft_consensus.cc:740] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21124dcbd9914973a616d9ed69870d0c, State: Initialized, Role: FOLLOWER
08:00:05.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.464011  9667 consensus_queue.cc:260] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 4.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
08:00:05.479 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 15 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328, tablet=e1d28f54265f409eb6d7b87d41754485, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:00:05.481 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.8.29.129:44331
08:00:05.481 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

08:00:05.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.481956  8310 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskivGXg8/build/asan/bin/kudu pid 9276
08:00:05.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.498600  9667 ts_tablet_manager.cc:1434] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c: Time spent starting tablet: real 0.049s	user 0.031s	sys 0.003s
08:00:05.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.503788  9776 heartbeater.cc:344] Connected to a master server at 127.8.29.190:44257
08:00:05.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.504258  9776 heartbeater.cc:461] Registering TS with master...
08:00:05.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.504992  9775 heartbeater.cc:344] Connected to a master server at 127.8.29.188:37745
08:00:05.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.505152  9775 heartbeater.cc:461] Registering TS with master...
08:00:05.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.505553  9776 heartbeater.cc:507] Master 127.8.29.190:44257 requested a full tablet report, sending...
08:00:05.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.506289  9400 generic_service.cc:196] Checking for leaks (request via RPC)
08:00:05.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.507205  9775 heartbeater.cc:507] Master 127.8.29.188:37745 requested a full tablet report, sending...
08:00:05.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.509191  8495 ts_manager.cc:194] Re-registered known tserver with Master: 21124dcbd9914973a616d9ed69870d0c (127.8.29.130:37019)
08:00:05.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.511339  8495 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.8.29.130:53783
08:00:05.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.513567  8353 ts_manager.cc:194] Re-registered known tserver with Master: 21124dcbd9914973a616d9ed69870d0c (127.8.29.130:37019)
08:00:05.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.516055  9775 heartbeater.cc:499] Master 127.8.29.188:37745 was elected leader, sending a full tablet report...
08:00:05.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.517976  9777 heartbeater.cc:344] Connected to a master server at 127.8.29.189:39965
08:00:05.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.518157  9777 heartbeater.cc:461] Registering TS with master...
08:00:05.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.518543  9777 heartbeater.cc:507] Master 127.8.29.189:39965 requested a full tablet report, sending...
08:00:05.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.520445  8424 ts_manager.cc:194] Re-registered known tserver with Master: 21124dcbd9914973a616d9ed69870d0c (127.8.29.130:37019)
08:00:05.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:05.618268  8310 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskivGXg8/build/asan/bin/kudu with pid 9276
08:00:05.637 [DEBUG - kudu-nio-1] (Connection.java:239) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] handling channelInactive
08:00:05.637 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:00:05.637 [DEBUG - kudu-nio-1] (Connection.java:709) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] cleaning up while in state READY due to: connection closed
08:00:05.639 [DEBUG - Time-limited test] (Connection.java:748) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] connecting to peer
08:00:05.640 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
08:00:05.640 [DEBUG - kudu-nio-2] (Connection.java:709) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] cleaning up while in state CONNECTING due to: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
08:00:05.641 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
08:00:05.666 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
08:00:05.667 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019),44002126d86d4b42930c65d831be6728(127.8.29.131:37185),bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)[L]]}]
08:00:05.667 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=2, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=29), Traces: [0ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [2ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [3ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [24ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [28ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@722616452(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@854881d -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2e380e27 -> wakeup thread Time-limited test)) after lookup
08:00:05.667 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] connecting to peer
08:00:05.668 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
08:00:05.669 [DEBUG - kudu-nio-3] (Connection.java:709) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] cleaning up while in state CONNECTING due to: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
08:00:05.669 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
08:00:05.686 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
08:00:05.686 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019),44002126d86d4b42930c65d831be6728(127.8.29.131:37185),bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)[L]]}]
08:00:05.686 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=4, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=48), Traces: [0ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [2ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [3ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [24ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [28ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [29ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [31ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [31ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [44ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [48ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@722616452(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@854881d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2003602086 after retry RPC@424409946), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2e380e27 -> wakeup thread Time-limited test -> (continuation of Deferred@2003602086 after retry RPC@424409946))) after lookup
08:00:05.687 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] connecting to peer
08:00:05.688 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
08:00:05.688 [DEBUG - kudu-nio-4] (Connection.java:709) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] cleaning up while in state CONNECTING due to: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
08:00:05.689 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
08:00:05.706 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
08:00:05.706 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019),44002126d86d4b42930c65d831be6728(127.8.29.131:37185),bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)[L]]}]
08:00:05.707 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=6, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=69), Traces: [0ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [2ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [3ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [24ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [28ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [29ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [31ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [31ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [44ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [48ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [49ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [51ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [51ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [68ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@722616452(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@854881d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2003602086 after retry RPC@424409946) -> (continuation of Deferred@1213840630 after retry RPC@244191116), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2e380e27 -> wakeup thread Time-limited test -> (continuation of Deferred@2003602086 after retry RPC@424409946) -> (continuation of Deferred@1213840630 after retry RPC@244191116))) after lookup
08:00:05.708 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] connecting to peer
08:00:05.709 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
08:00:05.709 [DEBUG - kudu-nio-5] (Connection.java:709) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] cleaning up while in state CONNECTING due to: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
08:00:05.709 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
08:00:05.786 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
08:00:05.786 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019),44002126d86d4b42930c65d831be6728(127.8.29.131:37185),bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)[L]]}]
08:00:05.787 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=8, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=149), Traces: [0ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [2ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [3ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [24ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [28ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [29ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [31ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [31ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [44ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [48ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [49ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [51ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [51ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [68ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [69ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [71ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [71ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [144ms] refreshing cache from master, [144ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [148ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@722616452(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@854881d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2003602086 after retry RPC@424409946) -> (continuation of Deferred@1213840630 after retry RPC@244191116) -> (continuation of Deferred@1608590551 after retry RPC@560833224), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2e380e27 -> wakeup thread Time-limited test -> (continuation of Deferred@2003602086 after retry RPC@424409946) -> (continuation of Deferred@1213840630 after retry RPC@244191116) -> (continuation of Deferred@1608590551 after retry RPC@560833224))) after lookup
08:00:05.787 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] connecting to peer
08:00:05.789 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
08:00:05.789 [DEBUG - kudu-nio-6] (Connection.java:709) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] cleaning up while in state CONNECTING due to: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
08:00:05.789 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
08:00:05.985 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
08:00:05.986 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019),44002126d86d4b42930c65d831be6728(127.8.29.131:37185),bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)[L]]}]
08:00:05.986 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=10, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=348), Traces: [0ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [2ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [3ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [24ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [28ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [29ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [31ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [31ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [44ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [48ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [49ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [51ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [51ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [68ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [69ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [71ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [71ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [144ms] refreshing cache from master, [144ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [148ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [149ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [151ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [151ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [344ms] refreshing cache from master, [344ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [347ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@722616452(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@854881d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2003602086 after retry RPC@424409946) -> (continuation of Deferred@1213840630 after retry RPC@244191116) -> (continuation of Deferred@1608590551 after retry RPC@560833224) -> (continuation of Deferred@461501800 after retry RPC@1463185202), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2e380e27 -> wakeup thread Time-limited test -> (continuation of Deferred@2003602086 after retry RPC@424409946) -> (continuation of Deferred@1213840630 after retry RPC@244191116) -> (continuation of Deferred@1608590551 after retry RPC@560833224) -> (continuation of Deferred@461501800 after retry RPC@1463185202))) after lookup
08:00:05.987 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] connecting to peer
08:00:05.988 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
08:00:05.988 [DEBUG - kudu-nio-7] (Connection.java:709) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] cleaning up while in state CONNECTING due to: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
08:00:05.988 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
08:00:06.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:06.931596  9791 raft_consensus.cc:493] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader bcfe780000f44bb5ba6ab6347414c393)
08:00:06.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:06.931886  9791 raft_consensus.cc:515] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
08:00:06.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:06.933843  9791 leader_election.cc:290] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bcfe780000f44bb5ba6ab6347414c393 (127.8.29.129:44331), 21124dcbd9914973a616d9ed69870d0c (127.8.29.130:37019)
08:00:06.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:06.936944  9475 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.29.129:44331: connect: Connection refused (error 111)
08:00:06.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:06.938930  9475 leader_election.cc:336] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer bcfe780000f44bb5ba6ab6347414c393 (127.8.29.129:44331): Network error: Client connection negotiation failed: client connection to 127.8.29.129:44331: connect: Connection refused (error 111)
08:00:06.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:06.944994  9730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1d28f54265f409eb6d7b87d41754485" candidate_uuid: "44002126d86d4b42930c65d831be6728" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "21124dcbd9914973a616d9ed69870d0c" is_pre_election: true
08:00:06.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:06.945792  9730 raft_consensus.cc:2468] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44002126d86d4b42930c65d831be6728 in term 4.
08:00:06.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:06.947091  9472 leader_election.cc:304] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 21124dcbd9914973a616d9ed69870d0c, 44002126d86d4b42930c65d831be6728; no voters: bcfe780000f44bb5ba6ab6347414c393
08:00:06.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:06.947669  9791 raft_consensus.cc:2804] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 5 FOLLOWER]: Leader pre-election won for term 6
08:00:06.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:06.947811  9791 raft_consensus.cc:493] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 5 FOLLOWER]: Starting leader election (detected failure of leader bcfe780000f44bb5ba6ab6347414c393)
08:00:06.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:06.947890  9791 raft_consensus.cc:3060] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 5 FOLLOWER]: Advancing to term 6
08:00:06.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:06.950933  9791 raft_consensus.cc:515] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
08:00:06.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:06.951792  9791 leader_election.cc:290] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [CANDIDATE]: Term 6 election: Requested vote from peers bcfe780000f44bb5ba6ab6347414c393 (127.8.29.129:44331), 21124dcbd9914973a616d9ed69870d0c (127.8.29.130:37019)
08:00:06.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:06.952402  9730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1d28f54265f409eb6d7b87d41754485" candidate_uuid: "44002126d86d4b42930c65d831be6728" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "21124dcbd9914973a616d9ed69870d0c"
08:00:06.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:06.952605  9730 raft_consensus.cc:3060] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 4 FOLLOWER]: Advancing to term 6
08:00:06.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:06.953969  9475 leader_election.cc:336] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer bcfe780000f44bb5ba6ab6347414c393 (127.8.29.129:44331): Network error: Client connection negotiation failed: client connection to 127.8.29.129:44331: connect: Connection refused (error 111)
08:00:06.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:06.956848  9730 raft_consensus.cc:2468] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44002126d86d4b42930c65d831be6728 in term 6.
08:00:06.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:06.957540  9472 leader_election.cc:304] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 21124dcbd9914973a616d9ed69870d0c, 44002126d86d4b42930c65d831be6728; no voters: bcfe780000f44bb5ba6ab6347414c393
08:00:06.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:06.957953  9791 raft_consensus.cc:2804] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 6 FOLLOWER]: Leader election won for term 6
08:00:06.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:06.958462  9791 raft_consensus.cc:697] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 6 LEADER]: Becoming Leader. State: Replica: 44002126d86d4b42930c65d831be6728, State: Running, Role: LEADER
08:00:06.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:06.959007  9791 consensus_queue.cc:237] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 5.20, Last appended by leader: 20, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
08:00:06.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:06.963253  8495 catalog_manager.cc:5696] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 reported cstate change: term changed from 5 to 6, leader changed from bcfe780000f44bb5ba6ab6347414c393 (127.8.29.129) to 44002126d86d4b42930c65d831be6728 (127.8.29.131). New cstate: current_term: 6 leader_uuid: "44002126d86d4b42930c65d831be6728" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } health_report { overall_health: HEALTHY } } }
08:00:07.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:07.379066  9475 consensus_peers.cc:597] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 -> Peer bcfe780000f44bb5ba6ab6347414c393 (127.8.29.129:44331): Couldn't send request to peer bcfe780000f44bb5ba6ab6347414c393. Status: Network error: Client connection negotiation failed: client connection to 127.8.29.129:44331: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
08:00:07.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:07.505041  9730 raft_consensus.cc:1275] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 6 FOLLOWER]: Refusing update from remote peer 44002126d86d4b42930c65d831be6728: Log matching property violated. Preceding OpId in replica: term: 4 index: 16. Preceding OpId from leader: term: 6 index: 21. (index mismatch)
08:00:07.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:07.506245  9791 consensus_queue.cc:1048] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 16, Time since last communication: 0.000s
08:00:07.745 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
08:00:07.746 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019),44002126d86d4b42930c65d831be6728(127.8.29.131:37185)[L],bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)]}]
08:00:07.746 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=12, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=2108), Traces: [0ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [2ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [3ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [24ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [28ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [29ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [31ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [31ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [44ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [48ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [49ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [51ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [51ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [68ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [69ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [71ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [71ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [144ms] refreshing cache from master, [144ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [148ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [149ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [151ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [151ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [344ms] refreshing cache from master, [344ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [347ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [348ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [350ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [350ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [2104ms] refreshing cache from master, [2104ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [2107ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@722616452(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@854881d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2003602086 after retry RPC@424409946) -> (continuation of Deferred@1213840630 after retry RPC@244191116) -> (continuation of Deferred@1608590551 after retry RPC@560833224) -> (continuation of Deferred@461501800 after retry RPC@1463185202) -> (continuation of Deferred@895490527 after retry RPC@1363918782), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2e380e27 -> wakeup thread Time-limited test -> (continuation of Deferred@2003602086 after retry RPC@424409946) -> (continuation of Deferred@1213840630 after retry RPC@244191116) -> (continuation of Deferred@1608590551 after retry RPC@560833224) -> (continuation of Deferred@461501800 after retry RPC@1463185202) -> (continuation of Deferred@895490527 after retry RPC@1363918782))) after lookup
08:00:07.747 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] connecting to peer
08:00:07.748 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] Successfully connected to peer
08:00:07.768 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x3a5f9759, L:/127.0.0.1:48354 - R:/127.8.29.131:37185] using TOKEN/null
08:00:07.798 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.8.29.129:44331
08:00:07.800 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

08:00:07.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:07.800724  8310 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskivGXg8/build/asan/bin/kudu
08:00:07.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskivGXg8/build/asan/bin/kudu
08:00:07.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/wal
08:00:07.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/data
08:00:07.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:00:07.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:00:07.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:00:07.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:00:07.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:00:07.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:00:07.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/logs
08:00:07.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/data/info.pb
08:00:07.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:00:07.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:00:07.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:00:07.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:00:07.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:00:07.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:00:07.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:00:07.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:00:07.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:00:07.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:00:07.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.8.29.129:44331
08:00:07.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.8.29.129
08:00:07.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.8.29.190:44257,127.8.29.189:39965,127.8.29.188:37745
08:00:07.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=38159
08:00:07.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.8.29.129
08:00:07.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.8.29.148:45029
08:00:07.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:00:07.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:00:07.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:00:08.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:08.181407  9819 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:00:08.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:08.181835  9819 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:00:08.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:08.181952  9819 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:00:08.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:08.191179  9819 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:00:08.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:08.191454  9819 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.129
08:00:08.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.201931  9819 tablet_server_runner.cc:78] Tablet server non-default flags:
08:00:08.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:00:08.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.8.29.148:45029
08:00:08.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:00:08.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:00:08.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/data
08:00:08.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/wal
08:00:08.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:00:08.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:00:08.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.8.29.129:44331
08:00:08.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:00:08.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:00:08.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:00:08.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/data/info.pb
08:00:08.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.8.29.129
08:00:08.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=38159
08:00:08.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.8.29.190:44257,127.8.29.189:39965,127.8.29.188:37745
08:00:08.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:00:08.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:00:08.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:00:08.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:00:08.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:00:08.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.8.29.129
08:00:08.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/logs
08:00:08.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:00:08.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:00:08.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:00:08.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:00:08.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:00:08.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
08:00:08.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:00:08.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 07:43:24 UTC on bdcb31816ec0
08:00:08.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11954
08:00:08.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
08:00:08.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.204192  9819 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:00:08.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.206133  9819 file_cache.cc:492] Constructed file cache file cache with capacity 419430
08:00:08.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:08.217831  9825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:00:08.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:08.217924  9824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:00:08.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:08.218080  9827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:00:08.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.219837  9819 server_base.cc:1061] running on GCE node
08:00:08.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.220670  9819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:00:08.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.221828  9819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:00:08.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.223070  9819 hybrid_clock.cc:648] HybridClock initialized: now 1779091208222999 us; error 64 us; skew 500 ppm
08:00:08.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.223523  9819 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:00:08.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.226034  9819 webserver.cc:533] Webserver started at http://127.8.29.129:38159/ using document root <none> and password file <none>
08:00:08.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.226809  9819 fs_manager.cc:362] Metadata directory not provided
08:00:08.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.226913  9819 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:00:08.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.233618  9819 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.004s	sys 0.000s
08:00:08.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.237074  9833 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:00:08.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.238678  9819 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
08:00:08.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.238838  9819 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/data,/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/wal
08:00:08.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "bcfe780000f44bb5ba6ab6347414c393"
08:00:08.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 07:59:45 on dist-test-slave-c1kb"
08:00:08.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.239645  9819 fs_report.cc:389] FS layout report
08:00:08.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:00:08.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/wal
08:00:08.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/wal
08:00:08.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/data/data
08:00:08.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:00:08.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:00:08.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:00:08.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:00:08.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:00:08.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:00:08.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:00:08.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:00:08.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:00:08.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:00:08.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:00:08.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.279529  9819 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:00:08.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.280498  9819 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:00:08.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.280772  9819 kserver.cc:163] Server-wide thread pool size limit: 3276
08:00:08.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.281886  9819 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
08:00:08.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.284644  9840 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
08:00:08.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.291950  9819 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
08:00:08.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.292079  9819 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.008s	user 0.001s	sys 0.000s
08:00:08.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.292189  9819 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
08:00:08.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.295585  9819 ts_tablet_manager.cc:616] Registered 1 tablets
08:00:08.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.295779  9819 ts_tablet_manager.cc:595] Time spent register tablets: real 0.004s	user 0.004s	sys 0.000s
08:00:08.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.296430  9840 tablet_bootstrap.cc:492] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393: Bootstrap starting.
08:00:08.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.329962  9840 log.cc:826] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393: Log is configured to *not* fsync() on all Append() calls
08:00:08.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.353446  9819 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.129:44331
08:00:08.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.353530  9947 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.29.129:44331 every 8 connection(s)
08:00:08.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.355278  9819 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/data/info.pb
08:00:08.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.364465  8310 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskivGXg8/build/asan/bin/kudu as pid 9819
08:00:08.365 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:00:08.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.366488  9840 tablet_bootstrap.cc:492] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=20 ignored=0} inserts{seen=15 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
08:00:08.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.368546  9840 tablet_bootstrap.cc:492] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393: Bootstrap complete.
08:00:08.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.370558  9840 ts_tablet_manager.cc:1403] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393: Time spent bootstrapping tablet: real 0.075s	user 0.051s	sys 0.016s
08:00:08.401 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 18 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328, tablet=e1d28f54265f409eb6d7b87d41754485, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:00:08.402 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.8.29.131:37185
08:00:08.403 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

08:00:08.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.403584  8310 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskivGXg8/build/asan/bin/kudu pid 9457
08:00:08.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.405210  9948 heartbeater.cc:344] Connected to a master server at 127.8.29.188:37745
08:00:08.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.405673  9948 heartbeater.cc:461] Registering TS with master...
08:00:08.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.404769  9840 raft_consensus.cc:359] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
08:00:08.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.405932  9840 raft_consensus.cc:740] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcfe780000f44bb5ba6ab6347414c393, State: Initialized, Role: FOLLOWER
08:00:08.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.406953  9948 heartbeater.cc:507] Master 127.8.29.188:37745 requested a full tablet report, sending...
08:00:08.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.413194  9949 heartbeater.cc:344] Connected to a master server at 127.8.29.190:44257
08:00:08.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.413400  9949 heartbeater.cc:461] Registering TS with master...
08:00:08.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.414290  9840 consensus_queue.cc:260] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 20, Last appended: 5.20, Last appended by leader: 20, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
08:00:08.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.417026  9949 heartbeater.cc:507] Master 127.8.29.190:44257 requested a full tablet report, sending...
08:00:08.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.420469  8353 ts_manager.cc:194] Re-registered known tserver with Master: bcfe780000f44bb5ba6ab6347414c393 (127.8.29.129:44331)
08:00:08.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.420799  8495 ts_manager.cc:194] Re-registered known tserver with Master: bcfe780000f44bb5ba6ab6347414c393 (127.8.29.129:44331)
08:00:08.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.422857  8495 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.8.29.129:58109
08:00:08.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.426955  9948 heartbeater.cc:499] Master 127.8.29.188:37745 was elected leader, sending a full tablet report...
08:00:08.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.427529  9840 ts_tablet_manager.cc:1434] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393: Time spent starting tablet: real 0.057s	user 0.012s	sys 0.002s
08:00:08.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.428109  9950 heartbeater.cc:344] Connected to a master server at 127.8.29.189:39965
08:00:08.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.428272  9950 heartbeater.cc:461] Registering TS with master...
08:00:08.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.428617  9950 heartbeater.cc:507] Master 127.8.29.189:39965 requested a full tablet report, sending...
08:00:08.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.430007  8424 ts_manager.cc:194] Re-registered known tserver with Master: bcfe780000f44bb5ba6ab6347414c393 (127.8.29.129:44331)
08:00:08.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.440105  9582 generic_service.cc:196] Checking for leaks (request via RPC)
08:00:08.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:08.547911  8310 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskivGXg8/build/asan/bin/kudu with pid 9457
08:00:08.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:08.567472  8410 connection.cc:582] server connection from 127.8.29.131:58623 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
08:00:08.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:08.567929  8339 connection.cc:582] server connection from 127.8.29.131:59295 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
08:00:08.568 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] handling channelInactive
08:00:08.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:08.568279  8479 connection.cc:582] server connection from 127.8.29.131:60651 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
08:00:08.568 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] cleaning up while in state READY due to: connection closed
08:00:08.568 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:00:08.569 [DEBUG - Time-limited test] (Connection.java:748) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] connecting to peer
08:00:08.570 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
08:00:08.570 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] cleaning up while in state CONNECTING due to: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
08:00:08.571 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 44002126d86d4b42930c65d831be6728(127.8.29.131:37185) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
08:00:08.585 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
08:00:08.586 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019),44002126d86d4b42930c65d831be6728(127.8.29.131:37185)[L],bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)]}]
08:00:08.586 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=2, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=17), Traces: [0ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [2ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [2ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [16ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@302931289(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@16327a0c -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@76503ae6 -> wakeup thread Time-limited test)) after lookup
08:00:08.586 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] connecting to peer
08:00:08.588 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
08:00:08.588 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] cleaning up while in state CONNECTING due to: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
08:00:08.588 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 44002126d86d4b42930c65d831be6728(127.8.29.131:37185) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
08:00:08.606 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
08:00:08.606 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019),44002126d86d4b42930c65d831be6728(127.8.29.131:37185)[L],bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)]}]
08:00:08.607 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=4, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=37), Traces: [0ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [2ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [2ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [16ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [17ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [19ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [19ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [36ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@302931289(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@16327a0c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@247829484 after retry RPC@2041913304), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@76503ae6 -> wakeup thread Time-limited test -> (continuation of Deferred@247829484 after retry RPC@2041913304))) after lookup
08:00:08.607 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] connecting to peer
08:00:08.608 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
08:00:08.608 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] cleaning up while in state CONNECTING due to: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
08:00:08.609 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 44002126d86d4b42930c65d831be6728(127.8.29.131:37185) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
08:00:08.626 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
08:00:08.626 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019),44002126d86d4b42930c65d831be6728(127.8.29.131:37185)[L],bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)]}]
08:00:08.626 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=6, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=57), Traces: [0ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [2ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [2ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [16ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [17ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [19ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [19ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [36ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [38ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [40ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [40ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [53ms] refreshing cache from master, [53ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [56ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@302931289(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@16327a0c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@247829484 after retry RPC@2041913304) -> (continuation of Deferred@409958248 after retry RPC@1232466018), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@76503ae6 -> wakeup thread Time-limited test -> (continuation of Deferred@247829484 after retry RPC@2041913304) -> (continuation of Deferred@409958248 after retry RPC@1232466018))) after lookup
08:00:08.628 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] connecting to peer
08:00:08.629 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
08:00:08.630 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] cleaning up while in state CONNECTING due to: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
08:00:08.630 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 44002126d86d4b42930c65d831be6728(127.8.29.131:37185) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
08:00:08.745 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
08:00:08.746 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019),44002126d86d4b42930c65d831be6728(127.8.29.131:37185)[L],bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)]}]
08:00:08.746 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=8, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=177), Traces: [0ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [2ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [2ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [16ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [17ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [19ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [19ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [36ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [38ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [40ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [40ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [53ms] refreshing cache from master, [53ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [56ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [59ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [61ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [61ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [173ms] refreshing cache from master, [173ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [176ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@302931289(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@16327a0c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@247829484 after retry RPC@2041913304) -> (continuation of Deferred@409958248 after retry RPC@1232466018) -> (continuation of Deferred@1099006691 after retry RPC@655461457), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@76503ae6 -> wakeup thread Time-limited test -> (continuation of Deferred@247829484 after retry RPC@2041913304) -> (continuation of Deferred@409958248 after retry RPC@1232466018) -> (continuation of Deferred@1099006691 after retry RPC@655461457))) after lookup
08:00:08.747 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] connecting to peer
08:00:08.748 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
08:00:08.748 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] cleaning up while in state CONNECTING due to: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
08:00:08.748 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 44002126d86d4b42930c65d831be6728(127.8.29.131:37185) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
08:00:08.965 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
08:00:08.966 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019),44002126d86d4b42930c65d831be6728(127.8.29.131:37185)[L],bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)]}]
08:00:08.966 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=10, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=397), Traces: [0ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [2ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [2ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [16ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [17ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [19ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [19ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [36ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [38ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [40ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [40ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [53ms] refreshing cache from master, [53ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [56ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [59ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [61ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [61ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [173ms] refreshing cache from master, [173ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [176ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [177ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [179ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [179ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [393ms] refreshing cache from master, [393ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [396ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@302931289(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@16327a0c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@247829484 after retry RPC@2041913304) -> (continuation of Deferred@409958248 after retry RPC@1232466018) -> (continuation of Deferred@1099006691 after retry RPC@655461457) -> (continuation of Deferred@1058632465 after retry RPC@339809191), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@76503ae6 -> wakeup thread Time-limited test -> (continuation of Deferred@247829484 after retry RPC@2041913304) -> (continuation of Deferred@409958248 after retry RPC@1232466018) -> (continuation of Deferred@1099006691 after retry RPC@655461457) -> (continuation of Deferred@1058632465 after retry RPC@339809191))) after lookup
08:00:08.967 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] connecting to peer
08:00:08.968 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
08:00:08.968 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] cleaning up while in state CONNECTING due to: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
08:00:08.968 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 44002126d86d4b42930c65d831be6728(127.8.29.131:37185) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
08:00:09.725 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
08:00:09.725 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019),44002126d86d4b42930c65d831be6728(127.8.29.131:37185)[L],bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)]}]
08:00:09.726 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=12, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=1156), Traces: [0ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [2ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [2ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [16ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [17ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [19ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [19ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [36ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [38ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [40ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [40ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [53ms] refreshing cache from master, [53ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [56ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [59ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [61ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [61ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [173ms] refreshing cache from master, [173ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [176ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [177ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [179ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [179ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [393ms] refreshing cache from master, [393ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [396ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [397ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [399ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [400ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [1153ms] refreshing cache from master, [1153ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [1156ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@302931289(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@16327a0c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@247829484 after retry RPC@2041913304) -> (continuation of Deferred@409958248 after retry RPC@1232466018) -> (continuation of Deferred@1099006691 after retry RPC@655461457) -> (continuation of Deferred@1058632465 after retry RPC@339809191) -> (continuation of Deferred@888011334 after retry RPC@682429271), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@76503ae6 -> wakeup thread Time-limited test -> (continuation of Deferred@247829484 after retry RPC@2041913304) -> (continuation of Deferred@409958248 after retry RPC@1232466018) -> (continuation of Deferred@1099006691 after retry RPC@655461457) -> (continuation of Deferred@1058632465 after retry RPC@339809191) -> (continuation of Deferred@888011334 after retry RPC@682429271))) after lookup
08:00:09.727 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] connecting to peer
08:00:09.728 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
08:00:09.728 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] cleaning up while in state CONNECTING due to: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
08:00:09.728 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 44002126d86d4b42930c65d831be6728(127.8.29.131:37185) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
08:00:09.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:09.831827  9964 raft_consensus.cc:493] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 44002126d86d4b42930c65d831be6728)
08:00:09.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:09.832077  9964 raft_consensus.cc:515] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
08:00:09.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:09.834301  9964 leader_election.cc:290] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bcfe780000f44bb5ba6ab6347414c393 (127.8.29.129:44331), 44002126d86d4b42930c65d831be6728 (127.8.29.131:37185)
08:00:09.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:09.837059  9662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.29.131:37185: connect: Connection refused (error 111)
08:00:09.839 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:09.839567  9662 leader_election.cc:336] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 44002126d86d4b42930c65d831be6728 (127.8.29.131:37185): Network error: Client connection negotiation failed: client connection to 127.8.29.131:37185: connect: Connection refused (error 111)
08:00:09.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:09.845329  9903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1d28f54265f409eb6d7b87d41754485" candidate_uuid: "21124dcbd9914973a616d9ed69870d0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "bcfe780000f44bb5ba6ab6347414c393" is_pre_election: true
08:00:09.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:09.845942  9903 raft_consensus.cc:2468] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21124dcbd9914973a616d9ed69870d0c in term 5.
08:00:09.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:09.847117  9664 leader_election.cc:304] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 21124dcbd9914973a616d9ed69870d0c, bcfe780000f44bb5ba6ab6347414c393; no voters: 44002126d86d4b42930c65d831be6728
08:00:09.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:09.847664  9964 raft_consensus.cc:2804] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 6 FOLLOWER]: Leader pre-election won for term 7
08:00:09.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:09.847851  9964 raft_consensus.cc:493] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 6 FOLLOWER]: Starting leader election (detected failure of leader 44002126d86d4b42930c65d831be6728)
08:00:09.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:09.847970  9964 raft_consensus.cc:3060] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 6 FOLLOWER]: Advancing to term 7
08:00:09.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:09.850124  9964 raft_consensus.cc:515] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
08:00:09.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:09.850960  9964 leader_election.cc:290] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [CANDIDATE]: Term 7 election: Requested vote from peers bcfe780000f44bb5ba6ab6347414c393 (127.8.29.129:44331), 44002126d86d4b42930c65d831be6728 (127.8.29.131:37185)
08:00:09.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:09.851493  9903 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1d28f54265f409eb6d7b87d41754485" candidate_uuid: "21124dcbd9914973a616d9ed69870d0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "bcfe780000f44bb5ba6ab6347414c393"
08:00:09.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:09.851753  9903 raft_consensus.cc:3060] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 5 FOLLOWER]: Advancing to term 7
08:00:09.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:09.852690  9662 leader_election.cc:336] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 44002126d86d4b42930c65d831be6728 (127.8.29.131:37185): Network error: Client connection negotiation failed: client connection to 127.8.29.131:37185: connect: Connection refused (error 111)
08:00:09.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:09.857065  9903 raft_consensus.cc:2468] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21124dcbd9914973a616d9ed69870d0c in term 7.
08:00:09.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:09.857985  9664 leader_election.cc:304] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 21124dcbd9914973a616d9ed69870d0c, bcfe780000f44bb5ba6ab6347414c393; no voters: 44002126d86d4b42930c65d831be6728
08:00:09.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:09.858618  9964 raft_consensus.cc:2804] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 7 FOLLOWER]: Leader election won for term 7
08:00:09.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:09.859282  9964 raft_consensus.cc:697] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 7 LEADER]: Becoming Leader. State: Replica: 21124dcbd9914973a616d9ed69870d0c, State: Running, Role: LEADER
08:00:09.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:09.860514  9964 consensus_queue.cc:237] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 6.24, Last appended by leader: 24, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
08:00:09.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:09.865082  8495 catalog_manager.cc:5696] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c reported cstate change: term changed from 6 to 7, leader changed from 44002126d86d4b42930c65d831be6728 (127.8.29.131) to 21124dcbd9914973a616d9ed69870d0c (127.8.29.130). New cstate: current_term: 7 leader_uuid: "21124dcbd9914973a616d9ed69870d0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } health_report { overall_health: UNKNOWN } } }
08:00:10.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:10.284090  9662 consensus_peers.cc:597] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c -> Peer 44002126d86d4b42930c65d831be6728 (127.8.29.131:37185): Couldn't send request to peer 44002126d86d4b42930c65d831be6728. Status: Network error: Client connection negotiation failed: client connection to 127.8.29.131:37185: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
08:00:10.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:10.419803  9903 raft_consensus.cc:1275] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 7 FOLLOWER]: Refusing update from remote peer 21124dcbd9914973a616d9ed69870d0c: Log matching property violated. Preceding OpId in replica: term: 5 index: 20. Preceding OpId from leader: term: 7 index: 25. (index mismatch)
08:00:10.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:10.420826  9964 consensus_queue.cc:1048] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 20, Time since last communication: 0.000s
08:00:12.186 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
08:00:12.186 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)[L],44002126d86d4b42930c65d831be6728(127.8.29.131:37185),bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)]}]
08:00:12.186 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=14, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=3617), Traces: [0ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [2ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [2ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [16ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [17ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [19ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [19ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [36ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [38ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [40ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [40ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [53ms] refreshing cache from master, [53ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [56ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [59ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [61ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [61ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [173ms] refreshing cache from master, [173ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [176ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [177ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [179ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [179ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [393ms] refreshing cache from master, [393ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [396ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [397ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [399ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [400ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [1153ms] refreshing cache from master, [1153ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [1156ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [1157ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [1159ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [1159ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [3613ms] refreshing cache from master, [3613ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [3616ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@302931289(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@16327a0c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@247829484 after retry RPC@2041913304) -> (continuation of Deferred@409958248 after retry RPC@1232466018) -> (continuation of Deferred@1099006691 after retry RPC@655461457) -> (continuation of Deferred@1058632465 after retry RPC@339809191) -> (continuation of Deferred@888011334 after retry RPC@682429271) -> (continuation of Deferred@1201939636 after retry RPC@127105188), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@76503ae6 -> wakeup thread Time-limited test -> (continuation of Deferred@247829484 after retry RPC@2041913304) -> (continuation of Deferred@409958248 after retry RPC@1232466018) -> (continuation of Deferred@1099006691 after retry RPC@655461457) -> (continuation of Deferred@1058632465 after retry RPC@339809191) -> (continuation of Deferred@888011334 after retry RPC@682429271) -> (continuation of Deferred@1201939636 after retry RPC@127105188))) after lookup
08:00:12.187 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] connecting to peer
08:00:12.188 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] Successfully connected to peer
08:00:12.206 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0xd57eedaf, L:/127.0.0.1:38120 - R:/127.8.29.130:37019] using TOKEN/null
08:00:12.233 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.8.29.131:37185
08:00:12.234 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

08:00:12.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.234927  8310 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskivGXg8/build/asan/bin/kudu
08:00:12.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskivGXg8/build/asan/bin/kudu
08:00:12.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/wal
08:00:12.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/data
08:00:12.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:00:12.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:00:12.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:00:12.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:00:12.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:00:12.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:00:12.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/logs
08:00:12.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/data/info.pb
08:00:12.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:00:12.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:00:12.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:00:12.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:00:12.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:00:12.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:00:12.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:00:12.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:00:12.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:00:12.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:00:12.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.8.29.131:37185
08:00:12.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.8.29.131
08:00:12.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.8.29.190:44257,127.8.29.189:39965,127.8.29.188:37745
08:00:12.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42487
08:00:12.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.8.29.131
08:00:12.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.8.29.148:45029
08:00:12.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:00:12.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:00:12.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:00:12.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:12.600579 10003 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:00:12.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:12.600924 10003 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:00:12.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:12.601002 10003 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:00:12.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:12.610351 10003 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:00:12.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:12.610546 10003 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.131
08:00:12.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.622268 10003 tablet_server_runner.cc:78] Tablet server non-default flags:
08:00:12.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:00:12.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.8.29.148:45029
08:00:12.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:00:12.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:00:12.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/data
08:00:12.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/wal
08:00:12.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:00:12.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:00:12.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.8.29.131:37185
08:00:12.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:00:12.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:00:12.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:00:12.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/data/info.pb
08:00:12.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.8.29.131
08:00:12.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42487
08:00:12.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.8.29.190:44257,127.8.29.189:39965,127.8.29.188:37745
08:00:12.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:00:12.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:00:12.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:00:12.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:00:12.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:00:12.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.8.29.131
08:00:12.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/logs
08:00:12.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:00:12.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:00:12.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:00:12.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:00:12.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:00:12.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
08:00:12.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:00:12.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 07:43:24 UTC on bdcb31816ec0
08:00:12.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11954
08:00:12.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
08:00:12.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.623986 10003 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:00:12.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.626307 10003 file_cache.cc:492] Constructed file cache file cache with capacity 419430
08:00:12.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:12.637207 10011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:00:12.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:12.637235 10010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:00:12.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:12.637811 10013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:00:12.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.639132 10003 server_base.cc:1061] running on GCE node
08:00:12.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.640028 10003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:00:12.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.641106 10003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:00:12.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.642374 10003 hybrid_clock.cc:648] HybridClock initialized: now 1779091212642296 us; error 67 us; skew 500 ppm
08:00:12.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.642863 10003 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:00:12.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.645610 10003 webserver.cc:533] Webserver started at http://127.8.29.131:42487/ using document root <none> and password file <none>
08:00:12.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.646649 10003 fs_manager.cc:362] Metadata directory not provided
08:00:12.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.646800 10003 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:00:12.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.653591 10003 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.000s	sys 0.004s
08:00:12.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.657763 10019 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:00:12.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.659780 10003 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
08:00:12.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.659996 10003 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/data,/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/wal
08:00:12.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "44002126d86d4b42930c65d831be6728"
08:00:12.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 07:59:46 on dist-test-slave-c1kb"
08:00:12.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.660873 10003 fs_report.cc:389] FS layout report
08:00:12.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:00:12.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/wal
08:00:12.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/wal
08:00:12.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/data/data
08:00:12.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:00:12.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:00:12.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:00:12.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:00:12.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:00:12.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:00:12.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:00:12.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:00:12.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:00:12.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:00:12.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:00:12.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.686481 10003 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:00:12.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.687618 10003 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:00:12.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.688011 10003 kserver.cc:163] Server-wide thread pool size limit: 3276
08:00:12.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.689311 10003 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
08:00:12.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.692066 10026 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
08:00:12.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.699185 10003 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
08:00:12.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.699347 10003 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.008s	user 0.000s	sys 0.001s
08:00:12.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.699517 10003 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
08:00:12.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.702695 10003 ts_tablet_manager.cc:616] Registered 1 tablets
08:00:12.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.702853 10003 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.000s	sys 0.000s
08:00:12.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.703397 10026 tablet_bootstrap.cc:492] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728: Bootstrap starting.
08:00:12.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.734541 10026 log.cc:826] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728: Log is configured to *not* fsync() on all Append() calls
08:00:12.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.753549 10003 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.131:37185
08:00:12.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.753669 10133 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.29.131:37185 every 8 connection(s)
08:00:12.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.755152 10003 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/data/info.pb
08:00:12.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.756614  8310 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskivGXg8/build/asan/bin/kudu as pid 10003
08:00:12.757 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:00:12.780 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 21 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328, tablet=e1d28f54265f409eb6d7b87d41754485, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:00:12.781 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.8.29.130:37019
08:00:12.781 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

08:00:12.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.782020  8310 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskivGXg8/build/asan/bin/kudu pid 9644
08:00:12.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.793339 10134 heartbeater.cc:344] Connected to a master server at 127.8.29.188:37745
08:00:12.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.793866 10134 heartbeater.cc:461] Registering TS with master...
08:00:12.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.795120 10134 heartbeater.cc:507] Master 127.8.29.188:37745 requested a full tablet report, sending...
08:00:12.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.798025 10135 heartbeater.cc:344] Connected to a master server at 127.8.29.190:44257
08:00:12.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.798171 10135 heartbeater.cc:461] Registering TS with master...
08:00:12.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.798394  8495 ts_manager.cc:194] Re-registered known tserver with Master: 44002126d86d4b42930c65d831be6728 (127.8.29.131:37185)
08:00:12.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.798734 10135 heartbeater.cc:507] Master 127.8.29.190:44257 requested a full tablet report, sending...
08:00:12.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.799464 10026 tablet_bootstrap.cc:492] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728: Bootstrap replayed 1/1 log segments. Stats: ops{read=24 overwritten=0 applied=24 ignored=0} inserts{seen=18 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
08:00:12.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.799952  8495 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.8.29.131:40895
08:00:12.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.800128 10026 tablet_bootstrap.cc:492] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728: Bootstrap complete.
08:00:12.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.801558  8353 ts_manager.cc:194] Re-registered known tserver with Master: 44002126d86d4b42930c65d831be6728 (127.8.29.131:37185)
08:00:12.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.801684 10026 ts_tablet_manager.cc:1403] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728: Time spent bootstrapping tablet: real 0.099s	user 0.052s	sys 0.017s
08:00:12.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.803781 10136 heartbeater.cc:344] Connected to a master server at 127.8.29.189:39965
08:00:12.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.803954 10136 heartbeater.cc:461] Registering TS with master...
08:00:12.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.804370 10136 heartbeater.cc:507] Master 127.8.29.189:39965 requested a full tablet report, sending...
08:00:12.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.805205  9770 generic_service.cc:196] Checking for leaks (request via RPC)
08:00:12.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.807930  8424 ts_manager.cc:194] Re-registered known tserver with Master: 44002126d86d4b42930c65d831be6728 (127.8.29.131:37185)
08:00:12.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.807250 10026 raft_consensus.cc:359] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
08:00:12.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.808338 10026 raft_consensus.cc:740] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44002126d86d4b42930c65d831be6728, State: Initialized, Role: FOLLOWER
08:00:12.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.809186 10026 consensus_queue.cc:260] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 6.24, Last appended by leader: 24, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
08:00:12.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.810316 10134 heartbeater.cc:499] Master 127.8.29.188:37745 was elected leader, sending a full tablet report...
08:00:12.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.810995 10026 ts_tablet_manager.cc:1434] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728: Time spent starting tablet: real 0.009s	user 0.010s	sys 0.000s
08:00:12.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:12.909017  8310 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskivGXg8/build/asan/bin/kudu with pid 9644
08:00:12.928 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:00:12.928 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] handling channelInactive
08:00:12.928 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] cleaning up while in state READY due to: connection closed
08:00:12.929 [DEBUG - Time-limited test] (Connection.java:748) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] connecting to peer
08:00:12.930 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:12.930 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] cleaning up while in state CONNECTING due to: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:12.930 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:12.945 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
08:00:12.946 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)[L],44002126d86d4b42930c65d831be6728(127.8.29.131:37185),bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)]}]
08:00:12.946 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=2, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=17), Traces: [0ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [1ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [1ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [16ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@449277077(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@14c4816a -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3670c439 -> wakeup thread Time-limited test)) after lookup
08:00:12.947 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] connecting to peer
08:00:12.948 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:12.948 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] cleaning up while in state CONNECTING due to: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:12.948 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:12.965 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
08:00:12.965 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)[L],44002126d86d4b42930c65d831be6728(127.8.29.131:37185),bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)]}]
08:00:12.966 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=4, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=37), Traces: [0ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [1ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [1ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [16ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [18ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [19ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [20ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [36ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@449277077(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@14c4816a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@6013349 after retry RPC@809143451), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3670c439 -> wakeup thread Time-limited test -> (continuation of Deferred@6013349 after retry RPC@809143451))) after lookup
08:00:12.966 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] connecting to peer
08:00:12.967 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:12.967 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] cleaning up while in state CONNECTING due to: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:12.967 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:13.005 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
08:00:13.005 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)[L],44002126d86d4b42930c65d831be6728(127.8.29.131:37185),bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)]}]
08:00:13.005 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=6, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=77), Traces: [0ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [1ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [1ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [16ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [18ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [19ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [20ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [36ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [37ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [38ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [38ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [73ms] refreshing cache from master, [73ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [76ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@449277077(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@14c4816a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@6013349 after retry RPC@809143451) -> (continuation of Deferred@1554629415 after retry RPC@949211773), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3670c439 -> wakeup thread Time-limited test -> (continuation of Deferred@6013349 after retry RPC@809143451) -> (continuation of Deferred@1554629415 after retry RPC@949211773))) after lookup
08:00:13.006 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] connecting to peer
08:00:13.007 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:13.007 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] cleaning up while in state CONNECTING due to: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:13.007 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:13.085 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
08:00:13.086 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)[L],44002126d86d4b42930c65d831be6728(127.8.29.131:37185),bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)]}]
08:00:13.086 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=8, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=157), Traces: [0ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [1ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [1ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [16ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [18ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [19ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [20ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [36ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [37ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [38ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [38ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [73ms] refreshing cache from master, [73ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [76ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [77ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [78ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [78ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [153ms] refreshing cache from master, [153ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [156ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@449277077(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@14c4816a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@6013349 after retry RPC@809143451) -> (continuation of Deferred@1554629415 after retry RPC@949211773) -> (continuation of Deferred@1798762037 after retry RPC@1292945684), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3670c439 -> wakeup thread Time-limited test -> (continuation of Deferred@6013349 after retry RPC@809143451) -> (continuation of Deferred@1554629415 after retry RPC@949211773) -> (continuation of Deferred@1798762037 after retry RPC@1292945684))) after lookup
08:00:13.087 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] connecting to peer
08:00:13.089 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:13.089 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] cleaning up while in state CONNECTING due to: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:13.089 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:13.546 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
08:00:13.546 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)[L],44002126d86d4b42930c65d831be6728(127.8.29.131:37185),bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)]}]
08:00:13.546 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=10, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=617), Traces: [0ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [1ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [1ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [16ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [18ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [19ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [20ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [36ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [37ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [38ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [38ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [73ms] refreshing cache from master, [73ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [76ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [77ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [78ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [78ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [153ms] refreshing cache from master, [153ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [156ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [158ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [160ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [160ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [613ms] refreshing cache from master, [613ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [616ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@449277077(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@14c4816a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@6013349 after retry RPC@809143451) -> (continuation of Deferred@1554629415 after retry RPC@949211773) -> (continuation of Deferred@1798762037 after retry RPC@1292945684) -> (continuation of Deferred@1389665468 after retry RPC@432514971), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3670c439 -> wakeup thread Time-limited test -> (continuation of Deferred@6013349 after retry RPC@809143451) -> (continuation of Deferred@1554629415 after retry RPC@949211773) -> (continuation of Deferred@1798762037 after retry RPC@1292945684) -> (continuation of Deferred@1389665468 after retry RPC@432514971))) after lookup
08:00:13.547 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] connecting to peer
08:00:13.548 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:13.548 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] cleaning up while in state CONNECTING due to: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:13.548 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:14.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.173652 10149 raft_consensus.cc:493] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 21124dcbd9914973a616d9ed69870d0c)
08:00:14.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.173946 10149 raft_consensus.cc:515] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
08:00:14.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.175757 10149 leader_election.cc:290] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 21124dcbd9914973a616d9ed69870d0c (127.8.29.130:37019), 44002126d86d4b42930c65d831be6728 (127.8.29.131:37185)
08:00:14.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:14.177498  9834 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.29.130:37019: connect: Connection refused (error 111)
08:00:14.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:14.179837  9834 leader_election.cc:336] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 21124dcbd9914973a616d9ed69870d0c (127.8.29.130:37019): Network error: Client connection negotiation failed: client connection to 127.8.29.130:37019: connect: Connection refused (error 111)
08:00:14.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.186079 10089 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1d28f54265f409eb6d7b87d41754485" candidate_uuid: "bcfe780000f44bb5ba6ab6347414c393" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "44002126d86d4b42930c65d831be6728" is_pre_election: true
08:00:14.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.186682 10089 raft_consensus.cc:2468] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcfe780000f44bb5ba6ab6347414c393 in term 6.
08:00:14.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.188493  9835 leader_election.cc:304] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44002126d86d4b42930c65d831be6728, bcfe780000f44bb5ba6ab6347414c393; no voters: 21124dcbd9914973a616d9ed69870d0c
08:00:14.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.189309 10149 raft_consensus.cc:2804] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 7 FOLLOWER]: Leader pre-election won for term 8
08:00:14.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.189600 10149 raft_consensus.cc:493] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 21124dcbd9914973a616d9ed69870d0c)
08:00:14.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.189728 10149 raft_consensus.cc:3060] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 7 FOLLOWER]: Advancing to term 8
08:00:14.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.192030 10149 raft_consensus.cc:515] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
08:00:14.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.192761 10149 leader_election.cc:290] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [CANDIDATE]: Term 8 election: Requested vote from peers 21124dcbd9914973a616d9ed69870d0c (127.8.29.130:37019), 44002126d86d4b42930c65d831be6728 (127.8.29.131:37185)
08:00:14.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.193609 10089 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1d28f54265f409eb6d7b87d41754485" candidate_uuid: "bcfe780000f44bb5ba6ab6347414c393" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "44002126d86d4b42930c65d831be6728"
08:00:14.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.193940 10089 raft_consensus.cc:3060] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 6 FOLLOWER]: Advancing to term 8
08:00:14.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:14.194950  9834 leader_election.cc:336] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 21124dcbd9914973a616d9ed69870d0c (127.8.29.130:37019): Network error: Client connection negotiation failed: client connection to 127.8.29.130:37019: connect: Connection refused (error 111)
08:00:14.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.198331 10089 raft_consensus.cc:2468] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcfe780000f44bb5ba6ab6347414c393 in term 8.
08:00:14.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.198941  9835 leader_election.cc:304] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44002126d86d4b42930c65d831be6728, bcfe780000f44bb5ba6ab6347414c393; no voters: 21124dcbd9914973a616d9ed69870d0c
08:00:14.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.199376 10149 raft_consensus.cc:2804] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 8 FOLLOWER]: Leader election won for term 8
08:00:14.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.200042 10149 raft_consensus.cc:697] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 8 LEADER]: Becoming Leader. State: Replica: bcfe780000f44bb5ba6ab6347414c393, State: Running, Role: LEADER
08:00:14.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.200647 10149 consensus_queue.cc:237] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 7.28, Last appended by leader: 28, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
08:00:14.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.205386  8495 catalog_manager.cc:5696] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 reported cstate change: term changed from 7 to 8, leader changed from 21124dcbd9914973a616d9ed69870d0c (127.8.29.130) to bcfe780000f44bb5ba6ab6347414c393 (127.8.29.129). New cstate: current_term: 8 leader_uuid: "bcfe780000f44bb5ba6ab6347414c393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } health_report { overall_health: UNKNOWN } } }
08:00:14.286 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
08:00:14.286 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019),44002126d86d4b42930c65d831be6728(127.8.29.131:37185),bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)[L]]}]
08:00:14.287 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=12, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=1358), Traces: [0ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [1ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [1ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [16ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [18ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [19ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [20ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [36ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [37ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [38ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [38ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [73ms] refreshing cache from master, [73ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [76ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [77ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [78ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [78ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [153ms] refreshing cache from master, [153ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [156ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [158ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [160ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [160ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [613ms] refreshing cache from master, [613ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [616ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [618ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [619ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [619ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [1353ms] refreshing cache from master, [1353ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [1356ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@449277077(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@14c4816a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@6013349 after retry RPC@809143451) -> (continuation of Deferred@1554629415 after retry RPC@949211773) -> (continuation of Deferred@1798762037 after retry RPC@1292945684) -> (continuation of Deferred@1389665468 after retry RPC@432514971) -> (continuation of Deferred@1189353577 after retry RPC@1648067665), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3670c439 -> wakeup thread Time-limited test -> (continuation of Deferred@6013349 after retry RPC@809143451) -> (continuation of Deferred@1554629415 after retry RPC@949211773) -> (continuation of Deferred@1798762037 after retry RPC@1292945684) -> (continuation of Deferred@1389665468 after retry RPC@432514971) -> (continuation of Deferred@1189353577 after retry RPC@1648067665))) after lookup
08:00:14.288 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] connecting to peer
08:00:14.289 [DEBUG - kudu-nio-7] (Connection.java:757) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] Successfully connected to peer
08:00:14.323 [DEBUG - kudu-nio-7] (Negotiator.java:919) Authenticated connection [id: 0x30a63687, L:/127.0.0.1:41390 - R:/127.8.29.129:44331] using TOKEN/null
08:00:14.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.333818 10089 raft_consensus.cc:1275] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 8 FOLLOWER]: Refusing update from remote peer bcfe780000f44bb5ba6ab6347414c393: Log matching property violated. Preceding OpId in replica: term: 6 index: 24. Preceding OpId from leader: term: 8 index: 30. (index mismatch)
08:00:14.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.334784 10149 consensus_queue.cc:1048] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 24, Time since last communication: 0.000s
08:00:14.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:14.337178  9834 consensus_peers.cc:597] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 -> Peer 21124dcbd9914973a616d9ed69870d0c (127.8.29.130:37019): Couldn't send request to peer 21124dcbd9914973a616d9ed69870d0c. Status: Network error: Client connection negotiation failed: client connection to 127.8.29.130:37019: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
08:00:14.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.349795 10168 mvcc.cc:204] Tried to move back new op lower bound from 7287157613899546624 to 7287157613371641856. Current Snapshot: MvccSnapshot[applied={T|T < 7287157613899546624}]
08:00:14.376 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.8.29.130:37019
08:00:14.377 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

08:00:14.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.377594  8310 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskivGXg8/build/asan/bin/kudu
08:00:14.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskivGXg8/build/asan/bin/kudu
08:00:14.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/wal
08:00:14.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/data
08:00:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:00:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:00:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:00:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:00:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:00:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:00:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/logs
08:00:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/data/info.pb
08:00:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:00:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:00:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:00:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:00:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:00:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:00:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:00:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:00:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:00:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:00:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.8.29.130:37019
08:00:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.8.29.130
08:00:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.8.29.190:44257,127.8.29.189:39965,127.8.29.188:37745
08:00:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44565
08:00:14.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.8.29.130
08:00:14.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.8.29.148:45029
08:00:14.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:00:14.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:00:14.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:00:14.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:14.750058 10177 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:00:14.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:14.750401 10177 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:00:14.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:14.750478 10177 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:00:14.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:14.759675 10177 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:00:14.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:14.759859 10177 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.130
08:00:14.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.771487 10177 tablet_server_runner.cc:78] Tablet server non-default flags:
08:00:14.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:00:14.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.8.29.148:45029
08:00:14.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:00:14.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:00:14.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/data
08:00:14.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/wal
08:00:14.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:00:14.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:00:14.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.8.29.130:37019
08:00:14.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:00:14.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:00:14.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:00:14.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/data/info.pb
08:00:14.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.8.29.130
08:00:14.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44565
08:00:14.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.8.29.190:44257,127.8.29.189:39965,127.8.29.188:37745
08:00:14.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:00:14.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:00:14.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:00:14.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:00:14.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:00:14.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.8.29.130
08:00:14.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/logs
08:00:14.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:00:14.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:00:14.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:00:14.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:00:14.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:00:14.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
08:00:14.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:00:14.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 07:43:24 UTC on bdcb31816ec0
08:00:14.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11954
08:00:14.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
08:00:14.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.773185 10177 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:00:14.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.775048 10177 file_cache.cc:492] Constructed file cache file cache with capacity 419430
08:00:14.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:14.788448 10182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:00:14.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:14.788795 10183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:00:14.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:14.789201 10185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:00:14.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.791314 10177 server_base.cc:1061] running on GCE node
08:00:14.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.792279 10177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:00:14.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.793581 10177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:00:14.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.794864 10177 hybrid_clock.cc:648] HybridClock initialized: now 1779091214794772 us; error 74 us; skew 500 ppm
08:00:14.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.795300 10177 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:00:14.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.798404 10177 webserver.cc:533] Webserver started at http://127.8.29.130:44565/ using document root <none> and password file <none>
08:00:14.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.799482 10177 fs_manager.cc:362] Metadata directory not provided
08:00:14.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.799666 10177 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:00:14.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.807657 10177 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.007s	sys 0.000s
08:00:14.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.811153 10191 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:00:14.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.812953 10177 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
08:00:14.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.813194 10177 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/data,/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/wal
08:00:14.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "21124dcbd9914973a616d9ed69870d0c"
08:00:14.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 07:59:46 on dist-test-slave-c1kb"
08:00:14.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.814177 10177 fs_report.cc:389] FS layout report
08:00:14.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:00:14.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/wal
08:00:14.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/wal
08:00:14.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/data/data
08:00:14.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:00:14.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:00:14.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:00:14.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:00:14.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:00:14.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:00:14.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:00:14.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:00:14.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:00:14.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:00:14.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:00:14.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.842829 10177 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:00:14.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.843997 10177 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:00:14.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.844364 10177 kserver.cc:163] Server-wide thread pool size limit: 3276
08:00:14.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.846117 10177 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
08:00:14.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.849689 10198 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
08:00:14.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.859148 10177 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
08:00:14.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.859306 10177 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.010s	user 0.000s	sys 0.001s
08:00:14.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.859659 10177 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
08:00:14.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.863595 10177 ts_tablet_manager.cc:616] Registered 1 tablets
08:00:14.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.863749 10177 ts_tablet_manager.cc:595] Time spent register tablets: real 0.004s	user 0.001s	sys 0.003s
08:00:14.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.864250 10198 tablet_bootstrap.cc:492] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c: Bootstrap starting.
08:00:14.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.897575 10198 log.cc:826] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c: Log is configured to *not* fsync() on all Append() calls
08:00:14.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.932530 10177 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.130:37019
08:00:14.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.932581 10305 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.29.130:37019 every 8 connection(s)
08:00:14.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.933926 10198 tablet_bootstrap.cc:492] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=21 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
08:00:14.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.934206 10177 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/data/info.pb
08:00:14.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.934900 10198 tablet_bootstrap.cc:492] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c: Bootstrap complete.
08:00:14.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.936259 10198 ts_tablet_manager.cc:1403] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c: Time spent bootstrapping tablet: real 0.072s	user 0.057s	sys 0.013s
08:00:14.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.939790  8310 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskivGXg8/build/asan/bin/kudu as pid 10177
08:00:14.940 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:00:14.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.944150 10198 raft_consensus.cc:359] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
08:00:14.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.945158 10198 raft_consensus.cc:740] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21124dcbd9914973a616d9ed69870d0c, State: Initialized, Role: FOLLOWER
08:00:14.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.946072 10198 consensus_queue.cc:260] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 7.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
08:00:14.972 [DEBUG - kudu-nio-7] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 24 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328, tablet=e1d28f54265f409eb6d7b87d41754485, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:00:14.975 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.8.29.129:44331
08:00:14.976 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

08:00:14.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.976661  8310 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskivGXg8/build/asan/bin/kudu pid 9819
08:00:14.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.998894 10306 heartbeater.cc:344] Connected to a master server at 127.8.29.188:37745
08:00:14.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:14.999359 10306 heartbeater.cc:461] Registering TS with master...
08:00:15.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:15.000667 10306 heartbeater.cc:507] Master 127.8.29.188:37745 requested a full tablet report, sending...
08:00:15.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:15.001238 10198 ts_tablet_manager.cc:1434] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c: Time spent starting tablet: real 0.065s	user 0.027s	sys 0.024s
08:00:15.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:15.003588 10308 heartbeater.cc:344] Connected to a master server at 127.8.29.189:39965
08:00:15.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:15.003855 10308 heartbeater.cc:461] Registering TS with master...
08:00:15.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:15.004504 10308 heartbeater.cc:507] Master 127.8.29.189:39965 requested a full tablet report, sending...
08:00:15.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:15.004921  8495 ts_manager.cc:194] Re-registered known tserver with Master: 21124dcbd9914973a616d9ed69870d0c (127.8.29.130:37019)
08:00:15.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:15.006681  8495 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.8.29.130:54461
08:00:15.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:15.008412 10307 heartbeater.cc:344] Connected to a master server at 127.8.29.190:44257
08:00:15.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:15.008554 10307 heartbeater.cc:461] Registering TS with master...
08:00:15.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:15.009346 10307 heartbeater.cc:507] Master 127.8.29.190:44257 requested a full tablet report, sending...
08:00:15.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:15.011317  8424 ts_manager.cc:194] Re-registered known tserver with Master: 21124dcbd9914973a616d9ed69870d0c (127.8.29.130:37019)
08:00:15.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:15.012521  9943 generic_service.cc:196] Checking for leaks (request via RPC)
08:00:15.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:15.013708  8353 ts_manager.cc:194] Re-registered known tserver with Master: 21124dcbd9914973a616d9ed69870d0c (127.8.29.130:37019)
08:00:15.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:15.016402 10306 heartbeater.cc:499] Master 127.8.29.188:37745 was elected leader, sending a full tablet report...
08:00:15.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:15.117311  8310 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskivGXg8/build/asan/bin/kudu with pid 9819
08:00:15.137 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:00:15.137 [DEBUG - kudu-nio-7] (Connection.java:239) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] handling channelInactive
08:00:15.137 [DEBUG - kudu-nio-7] (Connection.java:709) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] cleaning up while in state READY due to: connection closed
08:00:15.138 [DEBUG - Time-limited test] (Connection.java:748) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] connecting to peer
08:00:15.139 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
08:00:15.140 [DEBUG - kudu-nio-0] (Connection.java:709) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] cleaning up while in state CONNECTING due to: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
08:00:15.140 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
08:00:15.145 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
08:00:15.145 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019),44002126d86d4b42930c65d831be6728(127.8.29.131:37185),bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)[L]]}]
08:00:15.145 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=2, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=7), Traces: [0ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [2ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [2ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [3ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [7ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@1870636140(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@2e25b502 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@292bc7a -> wakeup thread Time-limited test)) after lookup
08:00:15.146 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] connecting to peer
08:00:15.147 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
08:00:15.148 [DEBUG - kudu-nio-1] (Connection.java:709) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] cleaning up while in state CONNECTING due to: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
08:00:15.148 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
08:00:15.165 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
08:00:15.166 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019),44002126d86d4b42930c65d831be6728(127.8.29.131:37185),bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)[L]]}]
08:00:15.166 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=4, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=28), Traces: [0ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [2ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [2ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [3ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [7ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [8ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [10ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [10ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [27ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@1870636140(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@2e25b502 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@963682269 after retry RPC@773232101), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@292bc7a -> wakeup thread Time-limited test -> (continuation of Deferred@963682269 after retry RPC@773232101))) after lookup
08:00:15.166 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] connecting to peer
08:00:15.168 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
08:00:15.168 [DEBUG - kudu-nio-2] (Connection.java:709) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] cleaning up while in state CONNECTING due to: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
08:00:15.168 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
08:00:15.206 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
08:00:15.206 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019),44002126d86d4b42930c65d831be6728(127.8.29.131:37185),bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)[L]]}]
08:00:15.206 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=6, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=68), Traces: [0ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [2ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [2ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [3ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [7ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [8ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [10ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [10ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [27ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [28ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [30ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [30ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [67ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@1870636140(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@2e25b502 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@963682269 after retry RPC@773232101) -> (continuation of Deferred@1969274424 after retry RPC@1875096780), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@292bc7a -> wakeup thread Time-limited test -> (continuation of Deferred@963682269 after retry RPC@773232101) -> (continuation of Deferred@1969274424 after retry RPC@1875096780))) after lookup
08:00:15.207 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] connecting to peer
08:00:15.208 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
08:00:15.208 [DEBUG - kudu-nio-3] (Connection.java:709) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] cleaning up while in state CONNECTING due to: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
08:00:15.208 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
08:00:15.327 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
08:00:15.327 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019),44002126d86d4b42930c65d831be6728(127.8.29.131:37185),bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)[L]]}]
08:00:15.328 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=8, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=190), Traces: [0ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [2ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [2ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [3ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [7ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [8ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [10ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [10ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [27ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [28ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [30ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [30ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [67ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [69ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [70ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [71ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [184ms] refreshing cache from master, [185ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [188ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@1870636140(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@2e25b502 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@963682269 after retry RPC@773232101) -> (continuation of Deferred@1969274424 after retry RPC@1875096780) -> (continuation of Deferred@1407465730 after retry RPC@569610854), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@292bc7a -> wakeup thread Time-limited test -> (continuation of Deferred@963682269 after retry RPC@773232101) -> (continuation of Deferred@1969274424 after retry RPC@1875096780) -> (continuation of Deferred@1407465730 after retry RPC@569610854))) after lookup
08:00:15.329 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] connecting to peer
08:00:15.330 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
08:00:15.330 [DEBUG - kudu-nio-4] (Connection.java:709) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] cleaning up while in state CONNECTING due to: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
08:00:15.330 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
08:00:15.766 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
08:00:15.766 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019),44002126d86d4b42930c65d831be6728(127.8.29.131:37185),bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)[L]]}]
08:00:15.767 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=10, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=628), Traces: [0ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [2ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [2ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [3ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [7ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [8ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [10ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [10ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [27ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [28ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [30ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [30ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [67ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [69ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [70ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [71ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [184ms] refreshing cache from master, [185ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [188ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [190ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [192ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [192ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [624ms] refreshing cache from master, [624ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [627ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@1870636140(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@2e25b502 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@963682269 after retry RPC@773232101) -> (continuation of Deferred@1969274424 after retry RPC@1875096780) -> (continuation of Deferred@1407465730 after retry RPC@569610854) -> (continuation of Deferred@739917199 after retry RPC@289929928), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@292bc7a -> wakeup thread Time-limited test -> (continuation of Deferred@963682269 after retry RPC@773232101) -> (continuation of Deferred@1969274424 after retry RPC@1875096780) -> (continuation of Deferred@1407465730 after retry RPC@569610854) -> (continuation of Deferred@739917199 after retry RPC@289929928))) after lookup
08:00:15.768 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] connecting to peer
08:00:15.769 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
08:00:15.769 [DEBUG - kudu-nio-5] (Connection.java:709) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] cleaning up while in state CONNECTING due to: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
08:00:15.769 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331
08:00:16.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:16.240061 10325 raft_consensus.cc:493] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
08:00:16.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:16.240326 10325 raft_consensus.cc:515] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
08:00:16.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:16.241946 10325 leader_election.cc:290] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bcfe780000f44bb5ba6ab6347414c393 (127.8.29.129:44331), 44002126d86d4b42930c65d831be6728 (127.8.29.131:37185)
08:00:16.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:16.244133 10195 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.29.129:44331: connect: Connection refused (error 111)
08:00:16.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:16.245750 10195 leader_election.cc:336] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer bcfe780000f44bb5ba6ab6347414c393 (127.8.29.129:44331): Network error: Client connection negotiation failed: client connection to 127.8.29.129:44331: connect: Connection refused (error 111)
08:00:16.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:16.249040 10089 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1d28f54265f409eb6d7b87d41754485" candidate_uuid: "21124dcbd9914973a616d9ed69870d0c" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "44002126d86d4b42930c65d831be6728" is_pre_election: true
08:00:16.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:16.249897 10193 leader_election.cc:304] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 21124dcbd9914973a616d9ed69870d0c; no voters: 44002126d86d4b42930c65d831be6728, bcfe780000f44bb5ba6ab6347414c393
08:00:16.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:16.250397 10325 raft_consensus.cc:2749] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
08:00:16.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:16.285952 10330 raft_consensus.cc:493] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader bcfe780000f44bb5ba6ab6347414c393)
08:00:16.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:16.286159 10330 raft_consensus.cc:515] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
08:00:16.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:16.287771 10330 leader_election.cc:290] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers bcfe780000f44bb5ba6ab6347414c393 (127.8.29.129:44331), 21124dcbd9914973a616d9ed69870d0c (127.8.29.130:37019)
08:00:16.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:16.289275 10023 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.29.129:44331: connect: Connection refused (error 111)
08:00:16.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:16.291047 10023 leader_election.cc:336] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer bcfe780000f44bb5ba6ab6347414c393 (127.8.29.129:44331): Network error: Client connection negotiation failed: client connection to 127.8.29.129:44331: connect: Connection refused (error 111)
08:00:16.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:16.295751 10261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1d28f54265f409eb6d7b87d41754485" candidate_uuid: "44002126d86d4b42930c65d831be6728" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "21124dcbd9914973a616d9ed69870d0c" is_pre_election: true
08:00:16.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:16.296139 10261 raft_consensus.cc:2468] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44002126d86d4b42930c65d831be6728 in term 7.
08:00:16.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:16.296859 10020 leader_election.cc:304] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 21124dcbd9914973a616d9ed69870d0c, 44002126d86d4b42930c65d831be6728; no voters: bcfe780000f44bb5ba6ab6347414c393
08:00:16.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:16.297303 10330 raft_consensus.cc:2804] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 8 FOLLOWER]: Leader pre-election won for term 9
08:00:16.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:16.297441 10330 raft_consensus.cc:493] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 8 FOLLOWER]: Starting leader election (detected failure of leader bcfe780000f44bb5ba6ab6347414c393)
08:00:16.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:16.297540 10330 raft_consensus.cc:3060] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 8 FOLLOWER]: Advancing to term 9
08:00:16.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:16.299635 10330 raft_consensus.cc:515] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
08:00:16.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:16.300318 10330 leader_election.cc:290] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [CANDIDATE]: Term 9 election: Requested vote from peers bcfe780000f44bb5ba6ab6347414c393 (127.8.29.129:44331), 21124dcbd9914973a616d9ed69870d0c (127.8.29.130:37019)
08:00:16.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:16.301003 10261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1d28f54265f409eb6d7b87d41754485" candidate_uuid: "44002126d86d4b42930c65d831be6728" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "21124dcbd9914973a616d9ed69870d0c"
08:00:16.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:16.301255 10261 raft_consensus.cc:3060] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 7 FOLLOWER]: Advancing to term 9
08:00:16.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:16.302297 10023 leader_election.cc:336] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer bcfe780000f44bb5ba6ab6347414c393 (127.8.29.129:44331): Network error: Client connection negotiation failed: client connection to 127.8.29.129:44331: connect: Connection refused (error 111)
08:00:16.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:16.305320 10261 raft_consensus.cc:2468] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44002126d86d4b42930c65d831be6728 in term 9.
08:00:16.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:16.305904 10020 leader_election.cc:304] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 21124dcbd9914973a616d9ed69870d0c, 44002126d86d4b42930c65d831be6728; no voters: bcfe780000f44bb5ba6ab6347414c393
08:00:16.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:16.306324 10330 raft_consensus.cc:2804] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 9 FOLLOWER]: Leader election won for term 9
08:00:16.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:16.306883 10330 raft_consensus.cc:697] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 9 LEADER]: Becoming Leader. State: Replica: 44002126d86d4b42930c65d831be6728, State: Running, Role: LEADER
08:00:16.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:16.307797 10330 consensus_queue.cc:237] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 8.32, Last appended by leader: 32, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
08:00:16.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:16.313032  8495 catalog_manager.cc:5696] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 reported cstate change: term changed from 8 to 9, leader changed from bcfe780000f44bb5ba6ab6347414c393 (127.8.29.129) to 44002126d86d4b42930c65d831be6728 (127.8.29.131). New cstate: current_term: 9 leader_uuid: "44002126d86d4b42930c65d831be6728" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } health_report { overall_health: HEALTHY } } }
08:00:16.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:16.809863 10023 consensus_peers.cc:597] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 -> Peer bcfe780000f44bb5ba6ab6347414c393 (127.8.29.129:44331): Couldn't send request to peer bcfe780000f44bb5ba6ab6347414c393. Status: Network error: Client connection negotiation failed: client connection to 127.8.29.129:44331: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
08:00:16.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:16.810444 10261 raft_consensus.cc:1275] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 9 FOLLOWER]: Refusing update from remote peer 44002126d86d4b42930c65d831be6728: Log matching property violated. Preceding OpId in replica: term: 7 index: 28. Preceding OpId from leader: term: 9 index: 33. (index mismatch)
08:00:16.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:16.811388 10330 consensus_queue.cc:1048] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 28, Time since last communication: 0.000s
08:00:16.865 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
08:00:16.865 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019),44002126d86d4b42930c65d831be6728(127.8.29.131:37185)[L],bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)]}]
08:00:16.865 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=12, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=1727), Traces: [0ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [2ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [2ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [3ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [7ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [8ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [10ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [10ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [27ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [28ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [30ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [30ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [67ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [69ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [70ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [71ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [184ms] refreshing cache from master, [185ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [188ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [190ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [192ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [192ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [624ms] refreshing cache from master, [624ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [627ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [629ms] sending RPC to server bcfe780000f44bb5ba6ab6347414c393, [631ms] received response from server bcfe780000f44bb5ba6ab6347414c393: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [631ms] delaying RPC due to: Network error: Failed to connect to peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331): Connection refused: /127.8.29.129:44331, [1723ms] refreshing cache from master, [1724ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [1727ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@1870636140(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@2e25b502 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@963682269 after retry RPC@773232101) -> (continuation of Deferred@1969274424 after retry RPC@1875096780) -> (continuation of Deferred@1407465730 after retry RPC@569610854) -> (continuation of Deferred@739917199 after retry RPC@289929928) -> (continuation of Deferred@779271415 after retry RPC@13468235), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@292bc7a -> wakeup thread Time-limited test -> (continuation of Deferred@963682269 after retry RPC@773232101) -> (continuation of Deferred@1969274424 after retry RPC@1875096780) -> (continuation of Deferred@1407465730 after retry RPC@569610854) -> (continuation of Deferred@739917199 after retry RPC@289929928) -> (continuation of Deferred@779271415 after retry RPC@13468235))) after lookup
08:00:16.866 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] connecting to peer
08:00:16.867 [DEBUG - kudu-nio-6] (Connection.java:757) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] Successfully connected to peer
08:00:16.925 [DEBUG - kudu-nio-6] (Negotiator.java:919) Authenticated connection [id: 0x7aa1fd9e, L:/127.0.0.1:60920 - R:/127.8.29.131:37185] using TOKEN/null
08:00:16.960 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.8.29.129:44331
08:00:16.960 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

08:00:16.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:16.961259  8310 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskivGXg8/build/asan/bin/kudu
08:00:16.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskivGXg8/build/asan/bin/kudu
08:00:16.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/wal
08:00:16.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/data
08:00:16.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:00:16.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:00:16.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:00:16.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:00:16.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:00:16.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:00:16.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/logs
08:00:16.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/data/info.pb
08:00:16.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:00:16.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:00:16.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:00:16.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:00:16.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:00:16.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:00:16.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:00:16.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:00:16.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:00:16.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:00:16.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.8.29.129:44331
08:00:16.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.8.29.129
08:00:16.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.8.29.190:44257,127.8.29.189:39965,127.8.29.188:37745
08:00:16.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=38159
08:00:16.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.8.29.129
08:00:16.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.8.29.148:45029
08:00:16.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:00:16.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:00:16.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:00:17.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:17.368738 10360 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:00:17.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:17.369057 10360 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:00:17.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:17.369131 10360 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:00:17.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:17.378696 10360 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:00:17.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:17.378880 10360 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.129
08:00:17.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.389436 10360 tablet_server_runner.cc:78] Tablet server non-default flags:
08:00:17.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:00:17.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.8.29.148:45029
08:00:17.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:00:17.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:00:17.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/data
08:00:17.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/wal
08:00:17.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:00:17.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:00:17.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.8.29.129:44331
08:00:17.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:00:17.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:00:17.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:00:17.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/data/info.pb
08:00:17.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.8.29.129
08:00:17.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=38159
08:00:17.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.8.29.190:44257,127.8.29.189:39965,127.8.29.188:37745
08:00:17.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:00:17.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:00:17.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:00:17.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:00:17.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:00:17.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.8.29.129
08:00:17.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/logs
08:00:17.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:00:17.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:00:17.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:00:17.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:00:17.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:00:17.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
08:00:17.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:00:17.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 07:43:24 UTC on bdcb31816ec0
08:00:17.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11954
08:00:17.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
08:00:17.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.391108 10360 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:00:17.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.393186 10360 file_cache.cc:492] Constructed file cache file cache with capacity 419430
08:00:17.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:17.403971 10368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:00:17.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:17.404022 10369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:00:17.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:17.404781 10371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:00:17.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.405273 10360 server_base.cc:1061] running on GCE node
08:00:17.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.405889 10360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:00:17.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.407052 10360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:00:17.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.408325 10360 hybrid_clock.cc:648] HybridClock initialized: now 1779091217408219 us; error 101 us; skew 500 ppm
08:00:17.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.408807 10360 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:00:17.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.413134 10360 webserver.cc:533] Webserver started at http://127.8.29.129:38159/ using document root <none> and password file <none>
08:00:17.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.413985 10360 fs_manager.cc:362] Metadata directory not provided
08:00:17.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.414124 10360 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:00:17.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.420982 10360 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.001s
08:00:17.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.424377 10377 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:00:17.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.426255 10360 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.000s
08:00:17.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.426461 10360 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/data,/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/wal
08:00:17.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "bcfe780000f44bb5ba6ab6347414c393"
08:00:17.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 07:59:45 on dist-test-slave-c1kb"
08:00:17.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.427268 10360 fs_report.cc:389] FS layout report
08:00:17.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:00:17.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/wal
08:00:17.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/wal
08:00:17.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/data/data
08:00:17.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:00:17.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:00:17.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:00:17.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:00:17.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:00:17.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:00:17.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:00:17.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:00:17.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:00:17.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:00:17.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:00:17.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.464226 10360 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:00:17.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.465220 10360 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:00:17.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.465569 10360 kserver.cc:163] Server-wide thread pool size limit: 3276
08:00:17.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.466631 10360 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
08:00:17.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.469165 10384 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
08:00:17.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.475349 10360 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
08:00:17.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.475544 10360 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.007s	user 0.001s	sys 0.000s
08:00:17.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.475793 10360 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
08:00:17.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.478947 10360 ts_tablet_manager.cc:616] Registered 1 tablets
08:00:17.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.479107 10360 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.000s	sys 0.000s
08:00:17.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.480118 10384 tablet_bootstrap.cc:492] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393: Bootstrap starting.
08:00:17.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.510898 10384 log.cc:826] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393: Log is configured to *not* fsync() on all Append() calls
08:00:17.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.535326 10360 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.129:44331
08:00:17.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.535370 10491 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.29.129:44331 every 8 connection(s)
08:00:17.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.536890 10360 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-0/data/info.pb
08:00:17.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.545351  8310 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskivGXg8/build/asan/bin/kudu as pid 10360
08:00:17.545 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:00:17.573 [DEBUG - kudu-nio-6] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 27 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328, tablet=e1d28f54265f409eb6d7b87d41754485, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:00:17.574 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.8.29.131:37185
08:00:17.575 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

08:00:17.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.575529  8310 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskivGXg8/build/asan/bin/kudu pid 10003
08:00:17.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.599131 10384 tablet_bootstrap.cc:492] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=24 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
08:00:17.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.599557 10492 heartbeater.cc:344] Connected to a master server at 127.8.29.188:37745
08:00:17.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.600018 10492 heartbeater.cc:461] Registering TS with master...
08:00:17.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.600562 10384 tablet_bootstrap.cc:492] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393: Bootstrap complete.
08:00:17.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.601339 10492 heartbeater.cc:507] Master 127.8.29.188:37745 requested a full tablet report, sending...
08:00:17.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.602376 10384 ts_tablet_manager.cc:1403] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393: Time spent bootstrapping tablet: real 0.123s	user 0.066s	sys 0.016s
08:00:17.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.604133 10129 generic_service.cc:196] Checking for leaks (request via RPC)
08:00:17.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.606315  8495 ts_manager.cc:194] Re-registered known tserver with Master: bcfe780000f44bb5ba6ab6347414c393 (127.8.29.129:44331)
08:00:17.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.607929  8495 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.8.29.129:58111
08:00:17.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.609292 10493 heartbeater.cc:344] Connected to a master server at 127.8.29.190:44257
08:00:17.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.609463 10493 heartbeater.cc:461] Registering TS with master...
08:00:17.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.610137 10493 heartbeater.cc:507] Master 127.8.29.190:44257 requested a full tablet report, sending...
08:00:17.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.610512 10384 raft_consensus.cc:359] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
08:00:17.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.611704 10384 raft_consensus.cc:740] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcfe780000f44bb5ba6ab6347414c393, State: Initialized, Role: FOLLOWER
08:00:17.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.611774  8353 ts_manager.cc:194] Re-registered known tserver with Master: bcfe780000f44bb5ba6ab6347414c393 (127.8.29.129:44331)
08:00:17.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.612612 10384 consensus_queue.cc:260] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 8.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
08:00:17.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.615701 10492 heartbeater.cc:499] Master 127.8.29.188:37745 was elected leader, sending a full tablet report...
08:00:17.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.616807 10384 ts_tablet_manager.cc:1434] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393: Time spent starting tablet: real 0.014s	user 0.013s	sys 0.000s
08:00:17.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.619382 10494 heartbeater.cc:344] Connected to a master server at 127.8.29.189:39965
08:00:17.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.619565 10494 heartbeater.cc:461] Registering TS with master...
08:00:17.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.619904 10494 heartbeater.cc:507] Master 127.8.29.189:39965 requested a full tablet report, sending...
08:00:17.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.621307  8424 ts_manager.cc:194] Re-registered known tserver with Master: bcfe780000f44bb5ba6ab6347414c393 (127.8.29.129:44331)
08:00:17.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:17.713383  8310 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskivGXg8/build/asan/bin/kudu with pid 10003
08:00:17.732 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:00:17.733 [DEBUG - kudu-nio-6] (Connection.java:239) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] handling channelInactive
08:00:17.733 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] cleaning up while in state READY due to: connection closed
08:00:17.733 [DEBUG - Time-limited test] (Connection.java:748) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] connecting to peer
08:00:17.734 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
08:00:17.734 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] cleaning up while in state CONNECTING due to: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
08:00:17.734 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 44002126d86d4b42930c65d831be6728(127.8.29.131:37185) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
08:00:17.745 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
08:00:17.746 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019),44002126d86d4b42930c65d831be6728(127.8.29.131:37185)[L],bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)]}]
08:00:17.746 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=2, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=13), Traces: [0ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [1ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [2ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [12ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@1156960325(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@28c6dd4a -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@508d2f8e -> wakeup thread Time-limited test)) after lookup
08:00:17.746 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] connecting to peer
08:00:17.747 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
08:00:17.747 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] cleaning up while in state CONNECTING due to: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
08:00:17.748 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 44002126d86d4b42930c65d831be6728(127.8.29.131:37185) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
08:00:17.766 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
08:00:17.766 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019),44002126d86d4b42930c65d831be6728(127.8.29.131:37185)[L],bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)]}]
08:00:17.766 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=4, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=33), Traces: [0ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [1ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [2ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [12ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [13ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [15ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [15ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [29ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [32ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@1156960325(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@28c6dd4a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1087141963 after retry RPC@1056298200), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@508d2f8e -> wakeup thread Time-limited test -> (continuation of Deferred@1087141963 after retry RPC@1056298200))) after lookup
08:00:17.767 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] connecting to peer
08:00:17.768 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
08:00:17.768 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] cleaning up while in state CONNECTING due to: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
08:00:17.768 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 44002126d86d4b42930c65d831be6728(127.8.29.131:37185) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
08:00:17.806 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
08:00:17.806 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019),44002126d86d4b42930c65d831be6728(127.8.29.131:37185)[L],bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)]}]
08:00:17.806 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=6, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=73), Traces: [0ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [1ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [2ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [12ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [13ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [15ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [15ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [29ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [32ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [34ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [35ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [35ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [69ms] refreshing cache from master, [69ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [72ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@1156960325(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@28c6dd4a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1087141963 after retry RPC@1056298200) -> (continuation of Deferred@1637600685 after retry RPC@1910019666), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@508d2f8e -> wakeup thread Time-limited test -> (continuation of Deferred@1087141963 after retry RPC@1056298200) -> (continuation of Deferred@1637600685 after retry RPC@1910019666))) after lookup
08:00:17.807 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] connecting to peer
08:00:17.808 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
08:00:17.808 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] cleaning up while in state CONNECTING due to: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
08:00:17.808 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 44002126d86d4b42930c65d831be6728(127.8.29.131:37185) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
08:00:17.906 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
08:00:17.907 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019),44002126d86d4b42930c65d831be6728(127.8.29.131:37185)[L],bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)]}]
08:00:17.907 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=8, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=174), Traces: [0ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [1ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [2ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [12ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [13ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [15ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [15ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [29ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [32ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [34ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [35ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [35ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [69ms] refreshing cache from master, [69ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [72ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [74ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [75ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [75ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [169ms] refreshing cache from master, [170ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [173ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@1156960325(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@28c6dd4a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1087141963 after retry RPC@1056298200) -> (continuation of Deferred@1637600685 after retry RPC@1910019666) -> (continuation of Deferred@1387692805 after retry RPC@1743385863), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@508d2f8e -> wakeup thread Time-limited test -> (continuation of Deferred@1087141963 after retry RPC@1056298200) -> (continuation of Deferred@1637600685 after retry RPC@1910019666) -> (continuation of Deferred@1387692805 after retry RPC@1743385863))) after lookup
08:00:17.908 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] connecting to peer
08:00:17.909 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
08:00:17.909 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] cleaning up while in state CONNECTING due to: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
08:00:17.909 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 44002126d86d4b42930c65d831be6728(127.8.29.131:37185) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
08:00:18.385 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
08:00:18.385 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019),44002126d86d4b42930c65d831be6728(127.8.29.131:37185)[L],bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)]}]
08:00:18.386 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=10, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=652), Traces: [0ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [1ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [2ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [12ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [13ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [15ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [15ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [29ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [32ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [34ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [35ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [35ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [69ms] refreshing cache from master, [69ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [72ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [74ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [75ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [75ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [169ms] refreshing cache from master, [170ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [173ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [175ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [176ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [176ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [649ms] refreshing cache from master, [649ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [652ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@1156960325(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@28c6dd4a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1087141963 after retry RPC@1056298200) -> (continuation of Deferred@1637600685 after retry RPC@1910019666) -> (continuation of Deferred@1387692805 after retry RPC@1743385863) -> (continuation of Deferred@144988281 after retry RPC@541967783), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@508d2f8e -> wakeup thread Time-limited test -> (continuation of Deferred@1087141963 after retry RPC@1056298200) -> (continuation of Deferred@1637600685 after retry RPC@1910019666) -> (continuation of Deferred@1387692805 after retry RPC@1743385863) -> (continuation of Deferred@144988281 after retry RPC@541967783))) after lookup
08:00:18.386 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] connecting to peer
08:00:18.387 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
08:00:18.387 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] cleaning up while in state CONNECTING due to: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
08:00:18.388 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 44002126d86d4b42930c65d831be6728(127.8.29.131:37185) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185
08:00:19.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:19.019578 10507 raft_consensus.cc:493] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 44002126d86d4b42930c65d831be6728)
08:00:19.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:19.019812 10507 raft_consensus.cc:515] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
08:00:19.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:19.020612 10507 leader_election.cc:290] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers bcfe780000f44bb5ba6ab6347414c393 (127.8.29.129:44331), 44002126d86d4b42930c65d831be6728 (127.8.29.131:37185)
08:00:19.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:19.023365 10193 leader_election.cc:336] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 44002126d86d4b42930c65d831be6728 (127.8.29.131:37185): Network error: Client connection negotiation failed: client connection to 127.8.29.131:37185: connect: Connection refused (error 111)
08:00:19.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:19.028963 10446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1d28f54265f409eb6d7b87d41754485" candidate_uuid: "21124dcbd9914973a616d9ed69870d0c" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "bcfe780000f44bb5ba6ab6347414c393" is_pre_election: true
08:00:19.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:19.029496 10446 raft_consensus.cc:2468] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21124dcbd9914973a616d9ed69870d0c in term 8.
08:00:19.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:19.030637 10195 leader_election.cc:304] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 21124dcbd9914973a616d9ed69870d0c, bcfe780000f44bb5ba6ab6347414c393; no voters: 44002126d86d4b42930c65d831be6728
08:00:19.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:19.031032 10507 raft_consensus.cc:2804] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 9 FOLLOWER]: Leader pre-election won for term 10
08:00:19.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:19.031157 10507 raft_consensus.cc:493] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 9 FOLLOWER]: Starting leader election (detected failure of leader 44002126d86d4b42930c65d831be6728)
08:00:19.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:19.031235 10507 raft_consensus.cc:3060] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 9 FOLLOWER]: Advancing to term 10
08:00:19.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:19.033291 10507 raft_consensus.cc:515] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
08:00:19.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:19.033893 10507 leader_election.cc:290] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [CANDIDATE]: Term 10 election: Requested vote from peers bcfe780000f44bb5ba6ab6347414c393 (127.8.29.129:44331), 44002126d86d4b42930c65d831be6728 (127.8.29.131:37185)
08:00:19.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:19.034399 10446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1d28f54265f409eb6d7b87d41754485" candidate_uuid: "21124dcbd9914973a616d9ed69870d0c" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "bcfe780000f44bb5ba6ab6347414c393"
08:00:19.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:19.034606 10446 raft_consensus.cc:3060] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 8 FOLLOWER]: Advancing to term 10
08:00:19.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:19.035560 10193 leader_election.cc:336] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 44002126d86d4b42930c65d831be6728 (127.8.29.131:37185): Network error: Client connection negotiation failed: client connection to 127.8.29.131:37185: connect: Connection refused (error 111)
08:00:19.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:19.038403 10446 raft_consensus.cc:2468] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21124dcbd9914973a616d9ed69870d0c in term 10.
08:00:19.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:19.038967 10195 leader_election.cc:304] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 21124dcbd9914973a616d9ed69870d0c, bcfe780000f44bb5ba6ab6347414c393; no voters: 44002126d86d4b42930c65d831be6728
08:00:19.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:19.039288 10507 raft_consensus.cc:2804] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 10 FOLLOWER]: Leader election won for term 10
08:00:19.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:19.039773 10507 raft_consensus.cc:697] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 10 LEADER]: Becoming Leader. State: Replica: 21124dcbd9914973a616d9ed69870d0c, State: Running, Role: LEADER
08:00:19.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:19.040290 10507 consensus_queue.cc:237] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 9.36, Last appended by leader: 36, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
08:00:19.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:19.044727  8495 catalog_manager.cc:5696] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c reported cstate change: term changed from 9 to 10, leader changed from 44002126d86d4b42930c65d831be6728 (127.8.29.131) to 21124dcbd9914973a616d9ed69870d0c (127.8.29.130). New cstate: current_term: 10 leader_uuid: "21124dcbd9914973a616d9ed69870d0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } health_report { overall_health: UNKNOWN } } }
08:00:19.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:19.548316 10446 raft_consensus.cc:1275] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 10 FOLLOWER]: Refusing update from remote peer 21124dcbd9914973a616d9ed69870d0c: Log matching property violated. Preceding OpId in replica: term: 8 index: 32. Preceding OpId from leader: term: 10 index: 37. (index mismatch)
08:00:19.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:19.549566 10524 consensus_queue.cc:1048] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 32, Time since last communication: 0.000s
08:00:19.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:19.567538 10193 consensus_peers.cc:597] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c -> Peer 44002126d86d4b42930c65d831be6728 (127.8.29.131:37185): Couldn't send request to peer 44002126d86d4b42930c65d831be6728. Status: Network error: Client connection negotiation failed: client connection to 127.8.29.131:37185: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
08:00:20.305 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
08:00:20.305 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)[L],44002126d86d4b42930c65d831be6728(127.8.29.131:37185),bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)]}]
08:00:20.305 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=12, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=2572), Traces: [0ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [1ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [2ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [9ms] refreshing cache from master, [9ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [12ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [13ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [15ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [15ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [29ms] refreshing cache from master, [30ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [32ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [34ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [35ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [35ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [69ms] refreshing cache from master, [69ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [72ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [74ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [75ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [75ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [169ms] refreshing cache from master, [170ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [173ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [175ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [176ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [176ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [649ms] refreshing cache from master, [649ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [652ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [653ms] sending RPC to server 44002126d86d4b42930c65d831be6728, [655ms] received response from server 44002126d86d4b42930c65d831be6728: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [655ms] delaying RPC due to: Network error: Failed to connect to peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185): Connection refused: /127.8.29.131:37185, [2569ms] refreshing cache from master, [2569ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [2572ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@1156960325(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@28c6dd4a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1087141963 after retry RPC@1056298200) -> (continuation of Deferred@1637600685 after retry RPC@1910019666) -> (continuation of Deferred@1387692805 after retry RPC@1743385863) -> (continuation of Deferred@144988281 after retry RPC@541967783) -> (continuation of Deferred@387637321 after retry RPC@1496608752), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@508d2f8e -> wakeup thread Time-limited test -> (continuation of Deferred@1087141963 after retry RPC@1056298200) -> (continuation of Deferred@1637600685 after retry RPC@1910019666) -> (continuation of Deferred@1387692805 after retry RPC@1743385863) -> (continuation of Deferred@144988281 after retry RPC@541967783) -> (continuation of Deferred@387637321 after retry RPC@1496608752))) after lookup
08:00:20.306 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] connecting to peer
08:00:20.307 [DEBUG - kudu-nio-5] (Connection.java:757) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] Successfully connected to peer
08:00:20.334 [DEBUG - kudu-nio-5] (Negotiator.java:919) Authenticated connection [id: 0x21ca2e47, L:/127.0.0.1:39422 - R:/127.8.29.130:37019] using TOKEN/null
08:00:20.369 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.8.29.131:37185
08:00:20.370 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

08:00:20.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.370826  8310 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskivGXg8/build/asan/bin/kudu
08:00:20.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskivGXg8/build/asan/bin/kudu
08:00:20.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/wal
08:00:20.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/data
08:00:20.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:00:20.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:00:20.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:00:20.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:00:20.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:00:20.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:00:20.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/logs
08:00:20.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/data/info.pb
08:00:20.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:00:20.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:00:20.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:00:20.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:00:20.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:00:20.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:00:20.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:00:20.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:00:20.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:00:20.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:00:20.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.8.29.131:37185
08:00:20.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.8.29.131
08:00:20.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.8.29.190:44257,127.8.29.189:39965,127.8.29.188:37745
08:00:20.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42487
08:00:20.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.8.29.131
08:00:20.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.8.29.148:45029
08:00:20.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:00:20.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:00:20.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:00:20.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:20.724647 10545 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:00:20.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:20.725032 10545 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:00:20.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:20.725137 10545 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:00:20.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:20.734045 10545 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:00:20.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:20.734299 10545 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.131
08:00:20.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.745260 10545 tablet_server_runner.cc:78] Tablet server non-default flags:
08:00:20.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:00:20.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.8.29.148:45029
08:00:20.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:00:20.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:00:20.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/data
08:00:20.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/wal
08:00:20.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:00:20.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:00:20.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.8.29.131:37185
08:00:20.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:00:20.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:00:20.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:00:20.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/data/info.pb
08:00:20.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.8.29.131
08:00:20.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42487
08:00:20.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.8.29.190:44257,127.8.29.189:39965,127.8.29.188:37745
08:00:20.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:00:20.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:00:20.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:00:20.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:00:20.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:00:20.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.8.29.131
08:00:20.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/logs
08:00:20.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:00:20.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:00:20.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:00:20.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:00:20.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:00:20.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
08:00:20.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:00:20.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 07:43:24 UTC on bdcb31816ec0
08:00:20.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11954
08:00:20.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
08:00:20.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.747402 10545 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:00:20.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.749414 10545 file_cache.cc:492] Constructed file cache file cache with capacity 419430
08:00:20.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:20.760267 10550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:00:20.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:20.760524 10553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:00:20.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.762097 10545 server_base.cc:1061] running on GCE node
08:00:20.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:20.760210 10551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:00:20.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.764158 10545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:00:20.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.765159 10545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:00:20.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.766422 10545 hybrid_clock.cc:648] HybridClock initialized: now 1779091220766359 us; error 61 us; skew 500 ppm
08:00:20.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.766712 10545 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:00:20.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.769013 10545 webserver.cc:533] Webserver started at http://127.8.29.131:42487/ using document root <none> and password file <none>
08:00:20.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.769847 10545 fs_manager.cc:362] Metadata directory not provided
08:00:20.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.769945 10545 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:00:20.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.775522 10545 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.001s
08:00:20.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.778306 10559 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:00:20.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.779783 10545 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
08:00:20.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.779964 10545 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/data,/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/wal
08:00:20.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "44002126d86d4b42930c65d831be6728"
08:00:20.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 07:59:46 on dist-test-slave-c1kb"
08:00:20.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.780691 10545 fs_report.cc:389] FS layout report
08:00:20.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:00:20.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/wal
08:00:20.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/wal
08:00:20.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/data/data
08:00:20.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:00:20.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:00:20.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:00:20.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:00:20.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:00:20.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:00:20.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:00:20.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:00:20.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:00:20.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:00:20.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:00:20.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.806861 10545 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:00:20.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.807926 10545 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:00:20.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.808275 10545 kserver.cc:163] Server-wide thread pool size limit: 3276
08:00:20.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.809357 10545 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
08:00:20.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.811861 10566 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
08:00:20.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.817903 10545 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
08:00:20.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.818076 10545 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.007s	user 0.000s	sys 0.001s
08:00:20.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.818220 10545 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
08:00:20.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.821738 10545 ts_tablet_manager.cc:616] Registered 1 tablets
08:00:20.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.821890 10545 ts_tablet_manager.cc:595] Time spent register tablets: real 0.004s	user 0.001s	sys 0.001s
08:00:20.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.822196 10566 tablet_bootstrap.cc:492] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728: Bootstrap starting.
08:00:20.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.853205 10566 log.cc:826] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728: Log is configured to *not* fsync() on all Append() calls
08:00:20.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.875089 10545 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.131:37185
08:00:20.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.875150 10673 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.29.131:37185 every 8 connection(s)
08:00:20.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.876958 10545 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-2/data/info.pb
08:00:20.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.881594  8310 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskivGXg8/build/asan/bin/kudu as pid 10545
08:00:20.882 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:00:20.914 [DEBUG - kudu-nio-5] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 30 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328, tablet=e1d28f54265f409eb6d7b87d41754485, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:00:20.915 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.8.29.130:37019
08:00:20.915 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

08:00:20.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.916222  8310 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskivGXg8/build/asan/bin/kudu pid 10177
08:00:20.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.917753 10566 tablet_bootstrap.cc:492] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728: Bootstrap replayed 1/1 log segments. Stats: ops{read=36 overwritten=0 applied=36 ignored=0} inserts{seen=27 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
08:00:20.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.918692 10566 tablet_bootstrap.cc:492] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728: Bootstrap complete.
08:00:20.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.920531 10566 ts_tablet_manager.cc:1403] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728: Time spent bootstrapping tablet: real 0.099s	user 0.065s	sys 0.028s
08:00:20.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.930229 10566 raft_consensus.cc:359] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 9 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
08:00:20.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.931329 10566 raft_consensus.cc:740] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44002126d86d4b42930c65d831be6728, State: Initialized, Role: FOLLOWER
08:00:20.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.934260 10301 generic_service.cc:196] Checking for leaks (request via RPC)
08:00:20.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.938362 10674 heartbeater.cc:344] Connected to a master server at 127.8.29.188:37745
08:00:20.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.938509 10675 heartbeater.cc:344] Connected to a master server at 127.8.29.190:44257
08:00:20.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.938482 10566 consensus_queue.cc:260] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 9.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
08:00:20.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.938989 10675 heartbeater.cc:461] Registering TS with master...
08:00:20.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.940408 10675 heartbeater.cc:507] Master 127.8.29.190:44257 requested a full tablet report, sending...
08:00:20.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.939013 10674 heartbeater.cc:461] Registering TS with master...
08:00:20.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.942565 10674 heartbeater.cc:507] Master 127.8.29.188:37745 requested a full tablet report, sending...
08:00:20.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.945029  8353 ts_manager.cc:194] Re-registered known tserver with Master: 44002126d86d4b42930c65d831be6728 (127.8.29.131:37185)
08:00:20.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.946261  8495 ts_manager.cc:194] Re-registered known tserver with Master: 44002126d86d4b42930c65d831be6728 (127.8.29.131:37185)
08:00:20.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.946605 10676 heartbeater.cc:344] Connected to a master server at 127.8.29.189:39965
08:00:20.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.946766 10676 heartbeater.cc:461] Registering TS with master...
08:00:20.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.947368 10676 heartbeater.cc:507] Master 127.8.29.189:39965 requested a full tablet report, sending...
08:00:20.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.947921  8495 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.8.29.131:52871
08:00:20.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.948807  8424 ts_manager.cc:194] Re-registered known tserver with Master: 44002126d86d4b42930c65d831be6728 (127.8.29.131:37185)
08:00:20.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.952229 10566 ts_tablet_manager.cc:1434] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728: Time spent starting tablet: real 0.032s	user 0.013s	sys 0.009s
08:00:20.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:20.953016 10674 heartbeater.cc:499] Master 127.8.29.188:37745 was elected leader, sending a full tablet report...
08:00:21.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:21.072314  8310 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskivGXg8/build/asan/bin/kudu with pid 10177
08:00:21.087 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:00:21.087 [DEBUG - kudu-nio-5] (Connection.java:239) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] handling channelInactive
08:00:21.087 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] cleaning up while in state READY due to: connection closed
08:00:21.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:21.088778 10690 negotiation.cc:336] Failed RPC negotiation. Trace:
08:00:21.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 08:00:21.056286 (+     0us) reactor.cc:678] Submitting negotiation task for server connection from 127.8.29.130:54501 (local address 127.8.29.131:37185)
08:00:21.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 08:00:21.076035 (+ 19749us) server_negotiation.cc:207] Beginning negotiation
08:00:21.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 08:00:21.076044 (+     9us) server_negotiation.cc:400] Waiting for connection header
08:00:21.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 08:00:21.076080 (+    36us) server_negotiation.cc:408] Connection header received
08:00:21.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 08:00:21.076181 (+   101us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
08:00:21.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 08:00:21.076188 (+     7us) server_negotiation.cc:462] Received NEGOTIATE request from client
08:00:21.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 08:00:21.076411 (+   223us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
08:00:21.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 08:00:21.076610 (+   199us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
08:00:21.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 08:00:21.087181 (+ 10571us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.130:54501: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
08:00:21.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":19578,"thread_start_us":19423,"threads_started":1}
08:00:21.103 [DEBUG - pool-2-thread-1] (Connection.java:748) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] connecting to peer
08:00:21.105 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:21.105 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] cleaning up while in state CONNECTING due to: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:21.106 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:21.125 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
08:00:21.125 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)[L],44002126d86d4b42930c65d831be6728(127.8.29.131:37185),bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)]}]
08:00:21.125 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=3, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=38), Traces: [0ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [1ms] delaying RPC due to: Illegal state: connection is terminated, [15ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [18ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [19ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [38ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@307424171(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@68634dfe -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@794bf554 -> wakeup thread Time-limited test)) after lookup
08:00:21.126 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] connecting to peer
08:00:21.127 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:21.127 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] cleaning up while in state CONNECTING due to: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:21.127 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:21.144 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
08:00:21.145 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)[L],44002126d86d4b42930c65d831be6728(127.8.29.131:37185),bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)]}]
08:00:21.145 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=5, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=57), Traces: [0ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [1ms] delaying RPC due to: Illegal state: connection is terminated, [15ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [18ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [19ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [38ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [39ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [40ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [40ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [55ms] refreshing cache from master, [55ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [57ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@307424171(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@68634dfe -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@238906517 after retry RPC@687524179), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@794bf554 -> wakeup thread Time-limited test -> (continuation of Deferred@238906517 after retry RPC@687524179))) after lookup
08:00:21.145 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] connecting to peer
08:00:21.146 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:21.146 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] cleaning up while in state CONNECTING due to: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:21.146 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:21.225 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
08:00:21.225 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)[L],44002126d86d4b42930c65d831be6728(127.8.29.131:37185),bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)]}]
08:00:21.226 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=7, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=138), Traces: [0ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [1ms] delaying RPC due to: Illegal state: connection is terminated, [15ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [18ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [19ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [38ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [39ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [40ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [40ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [55ms] refreshing cache from master, [55ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [57ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [58ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [59ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [59ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [134ms] refreshing cache from master, [135ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [138ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@307424171(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@68634dfe -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@238906517 after retry RPC@687524179) -> (continuation of Deferred@691465079 after retry RPC@444638345), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@794bf554 -> wakeup thread Time-limited test -> (continuation of Deferred@238906517 after retry RPC@687524179) -> (continuation of Deferred@691465079 after retry RPC@444638345))) after lookup
08:00:21.227 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] connecting to peer
08:00:21.228 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:21.229 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] cleaning up while in state CONNECTING due to: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:21.229 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:21.465 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
08:00:21.465 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)[L],44002126d86d4b42930c65d831be6728(127.8.29.131:37185),bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)]}]
08:00:21.465 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=9, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=378), Traces: [0ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [1ms] delaying RPC due to: Illegal state: connection is terminated, [15ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [18ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [19ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [38ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [39ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [40ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [40ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [55ms] refreshing cache from master, [55ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [57ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [58ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [59ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [59ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [134ms] refreshing cache from master, [135ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [138ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [140ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [142ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [142ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [375ms] refreshing cache from master, [375ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [378ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@307424171(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@68634dfe -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@238906517 after retry RPC@687524179) -> (continuation of Deferred@691465079 after retry RPC@444638345) -> (continuation of Deferred@220313540 after retry RPC@312147461), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@794bf554 -> wakeup thread Time-limited test -> (continuation of Deferred@238906517 after retry RPC@687524179) -> (continuation of Deferred@691465079 after retry RPC@444638345) -> (continuation of Deferred@220313540 after retry RPC@312147461))) after lookup
08:00:21.466 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] connecting to peer
08:00:21.467 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:21.467 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] cleaning up while in state CONNECTING due to: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:21.467 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019) for tablet e1d28f54265f409eb6d7b87d41754485: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019
08:00:21.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:21.870229 10691 raft_consensus.cc:493] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 21124dcbd9914973a616d9ed69870d0c)
08:00:21.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:21.870429 10691 raft_consensus.cc:515] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
08:00:21.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:21.871991 10691 leader_election.cc:290] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 21124dcbd9914973a616d9ed69870d0c (127.8.29.130:37019), 44002126d86d4b42930c65d831be6728 (127.8.29.131:37185)
08:00:21.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:21.873579 10378 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.29.130:37019: connect: Connection refused (error 111)
08:00:21.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:21.875276 10378 leader_election.cc:336] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 21124dcbd9914973a616d9ed69870d0c (127.8.29.130:37019): Network error: Client connection negotiation failed: client connection to 127.8.29.130:37019: connect: Connection refused (error 111)
08:00:21.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:21.880584 10628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1d28f54265f409eb6d7b87d41754485" candidate_uuid: "bcfe780000f44bb5ba6ab6347414c393" candidate_term: 11 candidate_status { last_received { term: 10 index: 40 } } ignore_live_leader: false dest_uuid: "44002126d86d4b42930c65d831be6728" is_pre_election: true
08:00:21.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:21.881114 10628 raft_consensus.cc:2468] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcfe780000f44bb5ba6ab6347414c393 in term 9.
08:00:21.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:21.882269 10379 leader_election.cc:304] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44002126d86d4b42930c65d831be6728, bcfe780000f44bb5ba6ab6347414c393; no voters: 21124dcbd9914973a616d9ed69870d0c
08:00:21.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:21.882778 10691 raft_consensus.cc:2804] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 10 FOLLOWER]: Leader pre-election won for term 11
08:00:21.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:21.882912 10691 raft_consensus.cc:493] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 21124dcbd9914973a616d9ed69870d0c)
08:00:21.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:21.883116 10691 raft_consensus.cc:3060] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 10 FOLLOWER]: Advancing to term 11
08:00:21.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:21.885205 10691 raft_consensus.cc:515] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
08:00:21.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:21.886018 10691 leader_election.cc:290] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [CANDIDATE]: Term 11 election: Requested vote from peers 21124dcbd9914973a616d9ed69870d0c (127.8.29.130:37019), 44002126d86d4b42930c65d831be6728 (127.8.29.131:37185)
08:00:21.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:21.886664 10628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1d28f54265f409eb6d7b87d41754485" candidate_uuid: "bcfe780000f44bb5ba6ab6347414c393" candidate_term: 11 candidate_status { last_received { term: 10 index: 40 } } ignore_live_leader: false dest_uuid: "44002126d86d4b42930c65d831be6728"
08:00:21.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:21.886905 10628 raft_consensus.cc:3060] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 9 FOLLOWER]: Advancing to term 11
08:00:21.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:21.887728 10378 leader_election.cc:336] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 21124dcbd9914973a616d9ed69870d0c (127.8.29.130:37019): Network error: Client connection negotiation failed: client connection to 127.8.29.130:37019: connect: Connection refused (error 111)
08:00:21.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:21.890823 10628 raft_consensus.cc:2468] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcfe780000f44bb5ba6ab6347414c393 in term 11.
08:00:21.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:21.891474 10379 leader_election.cc:304] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44002126d86d4b42930c65d831be6728, bcfe780000f44bb5ba6ab6347414c393; no voters: 21124dcbd9914973a616d9ed69870d0c
08:00:21.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:21.891847 10691 raft_consensus.cc:2804] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 11 FOLLOWER]: Leader election won for term 11
08:00:21.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:21.892397 10691 raft_consensus.cc:697] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [term 11 LEADER]: Becoming Leader. State: Replica: bcfe780000f44bb5ba6ab6347414c393, State: Running, Role: LEADER
08:00:21.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:21.892977 10691 consensus_queue.cc:237] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 10.40, Last appended by leader: 40, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
08:00:21.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:21.897024  8495 catalog_manager.cc:5696] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 reported cstate change: term changed from 10 to 11, leader changed from 21124dcbd9914973a616d9ed69870d0c (127.8.29.130) to bcfe780000f44bb5ba6ab6347414c393 (127.8.29.129). New cstate: current_term: 11 leader_uuid: "bcfe780000f44bb5ba6ab6347414c393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } health_report { overall_health: UNKNOWN } } }
08:00:22.045 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet e1d28f54265f409eb6d7b87d41754485 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328' with partition [<start>, <end>)
08:00:22.045 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=e1d28f54265f409eb6d7b87d41754485@[21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019),44002126d86d4b42930c65d831be6728(127.8.29.131:37185),bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)[L]]}]
08:00:22.046 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=e1d28f54265f409eb6d7b87d41754485, attempt=11, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=958), Traces: [0ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [1ms] delaying RPC due to: Illegal state: connection is terminated, [15ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [18ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [19ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [38ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [39ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [40ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [40ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [55ms] refreshing cache from master, [55ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [57ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [58ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [59ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [59ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [134ms] refreshing cache from master, [135ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [138ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [140ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [142ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [142ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [375ms] refreshing cache from master, [375ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [378ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, [379ms] sending RPC to server 21124dcbd9914973a616d9ed69870d0c, [380ms] received response from server 21124dcbd9914973a616d9ed69870d0c: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [381ms] delaying RPC due to: Network error: Failed to connect to peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019): Connection refused: /127.8.29.130:37019, [955ms] refreshing cache from master, [955ms] Sub RPC GetTableLocations: sending RPC to server master-127.8.29.188:37745, [958ms] Sub RPC GetTableLocations: received response from server master-127.8.29.188:37745: OK, deferred=Deferred@307424171(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$232/0x00007f89e8336730@68634dfe -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@238906517 after retry RPC@687524179) -> (continuation of Deferred@691465079 after retry RPC@444638345) -> (continuation of Deferred@220313540 after retry RPC@312147461) -> (continuation of Deferred@1996323337 after retry RPC@861063173), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@794bf554 -> wakeup thread Time-limited test -> (continuation of Deferred@238906517 after retry RPC@687524179) -> (continuation of Deferred@691465079 after retry RPC@444638345) -> (continuation of Deferred@220313540 after retry RPC@312147461) -> (continuation of Deferred@1996323337 after retry RPC@861063173))) after lookup
08:00:22.046 [DEBUG - kudu-nio-0] (Connection.java:748) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] connecting to peer
08:00:22.047 [DEBUG - kudu-nio-3] (Connection.java:757) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] Successfully connected to peer
08:00:22.065 [DEBUG - kudu-nio-3] (Negotiator.java:919) Authenticated connection [id: 0xba78df9f, L:/127.0.0.1:41454 - R:/127.8.29.129:44331] using TOKEN/null
08:00:22.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.070973 10628 raft_consensus.cc:1275] T e1d28f54265f409eb6d7b87d41754485 P 44002126d86d4b42930c65d831be6728 [term 11 FOLLOWER]: Refusing update from remote peer bcfe780000f44bb5ba6ab6347414c393: Log matching property violated. Preceding OpId in replica: term: 9 index: 36. Preceding OpId from leader: term: 11 index: 42. (index mismatch)
08:00:22.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:22.071728 10378 consensus_peers.cc:597] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 -> Peer 21124dcbd9914973a616d9ed69870d0c (127.8.29.130:37019): Couldn't send request to peer 21124dcbd9914973a616d9ed69870d0c. Status: Network error: Client connection negotiation failed: client connection to 127.8.29.130:37019: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
08:00:22.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.072072 10696 consensus_queue.cc:1048] T e1d28f54265f409eb6d7b87d41754485 P bcfe780000f44bb5ba6ab6347414c393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 36, Time since last communication: 0.000s
08:00:22.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.082432 10710 mvcc.cc:204] Tried to move back new op lower bound from 7287157645593141248 to 7287157644877594624. Current Snapshot: MvccSnapshot[applied={T|T < 7287157645593141248}]
08:00:22.105 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.8.29.130:37019
08:00:22.105 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

08:00:22.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.106387  8310 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskivGXg8/build/asan/bin/kudu
08:00:22.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskivGXg8/build/asan/bin/kudu
08:00:22.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/wal
08:00:22.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/data
08:00:22.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:00:22.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:00:22.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:00:22.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:00:22.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:00:22.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:00:22.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/logs
08:00:22.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/data/info.pb
08:00:22.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:00:22.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:00:22.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:00:22.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:00:22.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:00:22.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:00:22.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:00:22.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:00:22.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:00:22.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:00:22.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.8.29.130:37019
08:00:22.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.8.29.130
08:00:22.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.8.29.190:44257,127.8.29.189:39965,127.8.29.188:37745
08:00:22.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44565
08:00:22.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.8.29.130
08:00:22.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.8.29.148:45029
08:00:22.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:00:22.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:00:22.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:00:22.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:22.463624 10719 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:00:22.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:22.464047 10719 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:00:22.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:22.464164 10719 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:00:22.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:22.474673 10719 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:00:22.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:22.474924 10719 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.130
08:00:22.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.485525 10719 tablet_server_runner.cc:78] Tablet server non-default flags:
08:00:22.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:00:22.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.8.29.148:45029
08:00:22.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:00:22.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:00:22.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/data
08:00:22.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/wal
08:00:22.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:00:22.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:00:22.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.8.29.130:37019
08:00:22.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:00:22.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:00:22.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:00:22.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/data/info.pb
08:00:22.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.8.29.130
08:00:22.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44565
08:00:22.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.8.29.190:44257,127.8.29.189:39965,127.8.29.188:37745
08:00:22.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:00:22.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:00:22.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:00:22.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:00:22.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:00:22.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.8.29.130
08:00:22.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/logs
08:00:22.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:00:22.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:00:22.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:00:22.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:00:22.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:00:22.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
08:00:22.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:00:22.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 07:43:24 UTC on bdcb31816ec0
08:00:22.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11954
08:00:22.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
08:00:22.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.487877 10719 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:00:22.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.489811 10719 file_cache.cc:492] Constructed file cache file cache with capacity 419430
08:00:22.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:22.502939 10724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:00:22.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:22.503257 10725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:00:22.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:22.504243 10727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:00:22.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.505918 10719 server_base.cc:1061] running on GCE node
08:00:22.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.506817 10719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:00:22.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.508455 10719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:00:22.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.509735 10719 hybrid_clock.cc:648] HybridClock initialized: now 1779091222509688 us; error 51 us; skew 500 ppm
08:00:22.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.510162 10719 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:00:22.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.513423 10719 webserver.cc:533] Webserver started at http://127.8.29.130:44565/ using document root <none> and password file <none>
08:00:22.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.514492 10719 fs_manager.cc:362] Metadata directory not provided
08:00:22.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.514659 10719 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:00:22.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.522408 10719 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.005s	sys 0.000s
08:00:22.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.526698 10733 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:00:22.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.528403 10719 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
08:00:22.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.528551 10719 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/data,/tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/wal
08:00:22.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "21124dcbd9914973a616d9ed69870d0c"
08:00:22.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 07:59:46 on dist-test-slave-c1kb"
08:00:22.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.529268 10719 fs_report.cc:389] FS layout report
08:00:22.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:00:22.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/wal
08:00:22.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/wal
08:00:22.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/data/data
08:00:22.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:00:22.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:00:22.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:00:22.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:00:22.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:00:22.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:00:22.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:00:22.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:00:22.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:00:22.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:00:22.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:00:22.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.550124 10719 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:00:22.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.551229 10719 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:00:22.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.551568 10719 kserver.cc:163] Server-wide thread pool size limit: 3276
08:00:22.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.552770 10719 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
08:00:22.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.555636 10740 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
08:00:22.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.564085 10719 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
08:00:22.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.564242 10719 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.009s	user 0.001s	sys 0.000s
08:00:22.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.564388 10719 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
08:00:22.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.568090 10719 ts_tablet_manager.cc:616] Registered 1 tablets
08:00:22.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.568236 10719 ts_tablet_manager.cc:595] Time spent register tablets: real 0.004s	user 0.003s	sys 0.000s
08:00:22.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.568729 10740 tablet_bootstrap.cc:492] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c: Bootstrap starting.
08:00:22.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.601274 10740 log.cc:826] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c: Log is configured to *not* fsync() on all Append() calls
08:00:22.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.634680 10719 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.130:37019
08:00:22.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.634716 10847 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.29.130:37019 every 8 connection(s)
08:00:22.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.636416 10719 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskivGXg8/test-tmp/mini-kudu-cluster16627362922428117733/ts-1/data/info.pb
08:00:22.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.637480  8310 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskivGXg8/build/asan/bin/kudu as pid 10719
08:00:22.638 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:00:22.745 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328, tablet=e1d28f54265f409eb6d7b87d41754485, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:00:22.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.746517 10848 heartbeater.cc:344] Connected to a master server at 127.8.29.188:37745
08:00:22.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.747097 10848 heartbeater.cc:461] Registering TS with master...
08:00:22.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.749094 10848 heartbeater.cc:507] Master 127.8.29.188:37745 requested a full tablet report, sending...
08:00:22.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.752966  8495 ts_manager.cc:194] Re-registered known tserver with Master: 21124dcbd9914973a616d9ed69870d0c (127.8.29.130:37019)
08:00:22.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.754705  8495 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.8.29.130:42645
08:00:22.759 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779091187328, tablet=e1d28f54265f409eb6d7b87d41754485, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:00:22.766 [DEBUG - main] (Connection.java:575) [peer master-127.8.29.189:39965(127.8.29.189:39965)] disconnecting while in state READY
08:00:22.767 [DEBUG - main] (Connection.java:575) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] disconnecting while in state READY
08:00:22.769 [DEBUG - main] (Connection.java:575) [peer master-127.8.29.188:37745(127.8.29.188:37745)] disconnecting while in state READY
08:00:22.769 [DEBUG - main] (Connection.java:575) [peer 21124dcbd9914973a616d9ed69870d0c(127.8.29.130:37019)] disconnecting while in state TERMINATED
08:00:22.769 [DEBUG - main] (Connection.java:575) [peer master-127.8.29.190:44257(127.8.29.190:44257)] disconnecting while in state READY
08:00:22.769 [DEBUG - main] (Connection.java:575) [peer 44002126d86d4b42930c65d831be6728(127.8.29.131:37185)] disconnecting while in state TERMINATED
08:00:22.772 [DEBUG - kudu-nio-2] (Connection.java:239) [peer master-127.8.29.189:39965(127.8.29.189:39965)] handling channelInactive
08:00:22.773 [DEBUG - kudu-nio-2] (Connection.java:709) [peer master-127.8.29.189:39965(127.8.29.189:39965)] cleaning up while in state READY due to: connection closed
08:00:22.773 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.8.29.190:44257(127.8.29.190:44257)] handling channelInactive
08:00:22.773 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.8.29.190:44257(127.8.29.190:44257)] cleaning up while in state READY due to: connection closed
08:00:22.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.773957 10849 heartbeater.cc:344] Connected to a master server at 127.8.29.190:44257
08:00:22.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.774166 10849 heartbeater.cc:461] Registering TS with master...
08:00:22.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.774364 10853 heartbeater.cc:344] Connected to a master server at 127.8.29.189:39965
08:00:22.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.774554 10853 heartbeater.cc:461] Registering TS with master...
08:00:22.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.774796 10849 heartbeater.cc:507] Master 127.8.29.190:44257 requested a full tablet report, sending...
08:00:22.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.774897 10853 heartbeater.cc:507] Master 127.8.29.189:39965 requested a full tablet report, sending...
08:00:22.775 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.8.29.188:37745(127.8.29.188:37745)] handling channelInactive
08:00:22.775 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.8.29.188:37745(127.8.29.188:37745)] cleaning up while in state READY due to: connection closed
08:00:22.776 [DEBUG - kudu-nio-3] (AsyncKuduClient.java:2704) Releasing all remaining resources
08:00:22.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.776896  8424 ts_manager.cc:194] Re-registered known tserver with Master: 21124dcbd9914973a616d9ed69870d0c (127.8.29.130:37019)
08:00:22.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.778694  8353 ts_manager.cc:194] Re-registered known tserver with Master: 21124dcbd9914973a616d9ed69870d0c (127.8.29.130:37019)
08:00:22.779 [DEBUG - kudu-nio-3] (Connection.java:239) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] handling channelInactive
08:00:22.779 [DEBUG - kudu-nio-3] (Connection.java:709) [peer bcfe780000f44bb5ba6ab6347414c393(127.8.29.129:44331)] cleaning up while in state READY due to: connection closed
08:00:22.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.785449 10740 tablet_bootstrap.cc:492] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c: Bootstrap replayed 1/1 log segments. Stats: ops{read=40 overwritten=0 applied=40 ignored=0} inserts{seen=30 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
08:00:22.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.786513 10740 tablet_bootstrap.cc:492] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c: Bootstrap complete.
08:00:22.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.786857  8310 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskivGXg8/build/asan/bin/kudu pid 10360
08:00:22.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.792399 10740 ts_tablet_manager.cc:1403] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c: Time spent bootstrapping tablet: real 0.224s	user 0.059s	sys 0.031s
08:00:22.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.799772 10740 raft_consensus.cc:359] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 10 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
08:00:22.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.800800 10740 raft_consensus.cc:740] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [term 10 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21124dcbd9914973a616d9ed69870d0c, State: Initialized, Role: FOLLOWER
08:00:22.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.801662 10740 consensus_queue.cc:260] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 10.40, Last appended by leader: 40, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe780000f44bb5ba6ab6347414c393" member_type: VOTER last_known_addr { host: "127.8.29.129" port: 44331 } } peers { permanent_uuid: "21124dcbd9914973a616d9ed69870d0c" member_type: VOTER last_known_addr { host: "127.8.29.130" port: 37019 } } peers { permanent_uuid: "44002126d86d4b42930c65d831be6728" member_type: VOTER last_known_addr { host: "127.8.29.131" port: 37185 } }
08:00:22.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.802186 10487 generic_service.cc:196] Checking for leaks (request via RPC)
08:00:22.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.806945 10740 ts_tablet_manager.cc:1434] T e1d28f54265f409eb6d7b87d41754485 P 21124dcbd9914973a616d9ed69870d0c: Time spent starting tablet: real 0.013s	user 0.008s	sys 0.004s
08:00:22.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.809986 10848 heartbeater.cc:499] Master 127.8.29.188:37745 was elected leader, sending a full tablet report...
08:00:22.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.912659  8310 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskivGXg8/build/asan/bin/kudu with pid 10360
08:00:22.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.931654  8310 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskivGXg8/build/asan/bin/kudu pid 10719
08:00:22.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:22.940951 10843 generic_service.cc:196] Checking for leaks (request via RPC)
08:00:23.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:23.034790  8310 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskivGXg8/build/asan/bin/kudu with pid 10719
08:00:23.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:23.052326  8310 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskivGXg8/build/asan/bin/kudu pid 10545
08:00:23.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:23.059753 10669 generic_service.cc:196] Checking for leaks (request via RPC)
08:00:23.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:23.156486  8310 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskivGXg8/build/asan/bin/kudu with pid 10545
08:00:23.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:23.173878  8409 connection.cc:582] server connection from 127.8.29.131:47417 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
08:00:23.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:23.173928  8339 connection.cc:582] server connection from 127.8.29.131:38789 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
08:00:23.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 08:00:23.173928  8482 connection.cc:582] server connection from 127.8.29.131:52871 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
08:00:23.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:23.174322  8310 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskivGXg8/build/asan/bin/kudu pid 8322
08:00:23.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:23.175640  8383 generic_service.cc:196] Checking for leaks (request via RPC)
08:00:23.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:23.310729  8310 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskivGXg8/build/asan/bin/kudu with pid 8322
08:00:23.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:23.333504  8310 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskivGXg8/build/asan/bin/kudu pid 8390
08:00:23.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:23.334582  8454 generic_service.cc:196] Checking for leaks (request via RPC)
08:00:23.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:23.450771  8310 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskivGXg8/build/asan/bin/kudu with pid 8390
08:00:23.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:23.473119  8310 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskivGXg8/build/asan/bin/kudu pid 8461
08:00:23.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:23.474321  8525 generic_service.cc:196] Checking for leaks (request via RPC)
08:00:23.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 08:00:23.617534  8310 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskivGXg8/build/asan/bin/kudu with pid 8461
08:00:23.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-05-18T08:00:23Z chronyd exiting

Time: 41.646

OK (1 test)

