JUnit version 4.13.2
.08:16:15.898 [DEBUG - main] (RandomUtils.java:49) Using random seed: 1786522575891
08:16:15.952 [INFO - main] (RetryRule.java:138) Creating RetryStatement with result reporter and retry count of 0 (no retries)
08:16:16.008 [INFO - main] (KuduTestHarness.java:155) Creating a new MiniKuduCluster...
08:16:16.021 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-task0lHOBl/test-tmp
08:16:16.026 [INFO - main] (KuduBinaryLocator.java:83) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/asan/bin
08:16:16.028 [INFO - main] (MiniKuduCluster.java:217) Starting process: [/tmp/dist-test-task0lHOBl/java/build/dist-test/../../../build/asan/bin/kudu, test, mini_cluster, --serialization=pb]
08:16:16.216 [DEBUG - main] (MiniKuduCluster.java:185) Request: create_cluster {
  num_masters: 3
  num_tservers: 3
  enable_kerberos: false
  cluster_root: "/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426"
  hms_mode: NONE
  mini_kdc_options {
  }
  principal: "kudu"
  mini_oidc_options {
  }
}

08:16:16.462 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
08:16:16.463 [DEBUG - main] (MiniKuduCluster.java:185) Request: start_cluster {
}

08:16:16.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:16.467249 10401 mini_oidc.cc:150] Starting JWKS server
08:16:16.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:16.472023 10401 webserver.cc:533] Webserver started at http://127.0.0.1:34289/ using document root <none> and password file <none>
08:16:16.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:16.472414 10401 mini_oidc.cc:186] Starting OIDC Discovery server
08:16:16.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:16.473691 10401 webserver.cc:533] Webserver started at http://127.0.0.1:34229/ using document root <none> and password file <none>
08:16:16.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-08-12T08:16:16Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
08:16:16.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-08-12T08:16:16Z Disabled control of system clock
08:16:16.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:16.513969 10401 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0lHOBl/build/asan/bin/kudu
08:16:16.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task0lHOBl/build/asan/bin/kudu
08:16:16.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-0/wal
08:16:16.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-0/data
08:16:16.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:16:16.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:16:16.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:16:16.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:16:16.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:16:16.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:16:16.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-0/logs
08:16:16.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-0/data/info.pb
08:16:16.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:16:16.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:16:16.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:16:16.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:16:16.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:16:16.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:16:16.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:16:16.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:16:16.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
08:16:16.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:16:16.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
08:16:16.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
08:16:16.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.40.126:45793
08:16:16.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.40.126
08:16:16.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
08:16:16.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.40.84:40955
08:16:16.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:16:16.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:16:16.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:16:16.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
08:16:16.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.10.40.126:45793,127.10.40.125:46051,127.10.40.124:41773
08:16:16.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
08:16:16.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34289/jwks.json with env {}
08:16:16.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:16.872808 10414 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:16:16.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:16.873145 10414 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:16:16.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:16.873224 10414 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:16:16.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:16.882128 10414 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
08:16:16.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:16.882226 10414 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:16:16.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:16.882287 10414 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
08:16:16.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:16.882366 10414 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
08:16:16.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:16.882465 10414 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
08:16:16.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:16.882520 10414 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:34289/jwks.json
08:16:16.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:16.893743 10414 master_runner.cc:387] Master server non-default flags:
08:16:16.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-0/logs
08:16:16.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:16:16.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:16:16.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:16:16.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.40.84:40955
08:16:16.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:16:16.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:16:16.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-0/data
08:16:16.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-0/wal
08:16:16.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
08:16:16.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.10.40.126:45793,127.10.40.125:46051,127.10.40.124:41773
08:16:16.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:16:16.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:16:16.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
08:16:16.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.40.126:45793
08:16:16.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
08:16:16.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:16:16.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
08:16:16.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34289/jwks.json
08:16:16.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:16:16.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:16:16.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-0/data/info.pb
08:16:16.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.40.126
08:16:16.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
08:16:16.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:16:16.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:16:16.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:16:16.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:16:16.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:16:16.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:16:16.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
08:16:16.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:16:16.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 70c51312f1186a8ae9f46647160b7730123f7601
08:16:16.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:16:16.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 12 Aug 2026 07:43:21 UTC on bdcb31816ec0
08:16:16.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 13302
08:16:16.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
08:16:16.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:16.895784 10414 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:16:16.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:16.897742 10414 file_cache.cc:504] Constructed file cache file cache with capacity 419430
08:16:16.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:16.907193 10420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:16:16.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:16.907110 10419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:16:16.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:16.907783 10422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:16:16.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:16.909233 10414 server_base.cc:1061] running on GCE node
08:16:16.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:16.910236 10414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:16:16.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:16.911859 10414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:16:16.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:16.913100 10414 hybrid_clock.cc:648] HybridClock initialized: now 1786522576913088 us; error 64 us; skew 500 ppm
08:16:16.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:16.913579 10414 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:16:16.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:16.916011 10414 webserver.cc:533] Webserver started at http://127.10.40.126:34815/ using document root <none> and password file <none>
08:16:16.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:16.916867 10414 fs_manager.cc:362] Metadata directory not provided
08:16:16.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:16.917006 10414 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:16:16.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:16.917500 10414 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
08:16:16.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:16.920140 10414 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-0/data/instance:
08:16:16.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "a0b587403b6a4d3090cfced49037efc0"
08:16:16.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:16:16 on dist-test-slave-gp6n"
08:16:16.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:16.921108 10414 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-0/wal/instance:
08:16:16.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "a0b587403b6a4d3090cfced49037efc0"
08:16:16.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:16:16 on dist-test-slave-gp6n"
08:16:16.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:16.927022 10414 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.001s
08:16:16.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:16.930809 10429 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:16:16.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:16.932354 10414 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
08:16:16.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:16.932528 10414 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-0/data,/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-0/wal
08:16:16.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "a0b587403b6a4d3090cfced49037efc0"
08:16:16.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:16:16 on dist-test-slave-gp6n"
08:16:16.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:16.932708 10414 fs_report.cc:389] FS layout report
08:16:16.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:16:16.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-0/wal
08:16:16.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-0/wal
08:16:16.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-0/data/data
08:16:16.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:16:16.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:16:16.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:16:16.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:16:16.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:16:16.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:16:16.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:16:16.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:16:16.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:16:16.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:16:16.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:16:16.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:16.957743 10414 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:16:16.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:16.958699 10414 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:16:16.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:16.958946 10414 kserver.cc:163] Server-wide thread pool size limit: 3276
08:16:16.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:16.980177 10414 rpc_server.cc:307] RPC server started. Bound to: 127.10.40.126:45793
08:16:16.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:16.980186 10499 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.40.126:45793 every 8 connection(s)
08:16:16.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:16.982087 10414 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-0/data/info.pb
08:16:16.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:16.987234 10401 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0lHOBl/build/asan/bin/kudu as pid 10414
08:16:16.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:16.987576 10401 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-0/wal/instance
08:16:16.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:16.987677 10500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
08:16:16.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:16.990128 10401 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0lHOBl/build/asan/bin/kudu
08:16:16.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task0lHOBl/build/asan/bin/kudu
08:16:16.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-1/wal
08:16:16.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-1/data
08:16:16.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:16:16.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:16:16.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:16:16.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:16:16.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:16:16.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:16:16.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-1/logs
08:16:16.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-1/data/info.pb
08:16:16.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:16:16.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:16:16.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:16:16.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:16:16.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:16:16.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:16:16.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:16:16.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:16:16.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
08:16:16.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:16:16.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
08:16:16.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
08:16:16.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.40.125:46051
08:16:16.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.40.125
08:16:16.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
08:16:16.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.40.84:40955
08:16:16.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:16:16.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:16:16.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:16:16.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
08:16:16.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.10.40.126:45793,127.10.40.125:46051,127.10.40.124:41773
08:16:16.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
08:16:16.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34289/jwks.json with env {}
08:16:16.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:16.995441 10500 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.40.126" port: 45793 } has no permanent_uuid. Determining permanent_uuid...
08:16:17.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.013170 10500 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.40.125" port: 46051 } has no permanent_uuid. Determining permanent_uuid...
08:16:17.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:17.014232 10431 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.40.125:46051: connect: Connection refused (error 111)
08:16:17.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:17.015980 10500 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.10.40.125:46051: Network error: Client connection negotiation failed: client connection to 127.10.40.125:46051: connect: Connection refused (error 111)
08:16:17.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.065402 10500 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.40.125" port: 46051 } attempt: 1
08:16:17.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:17.067745 10500 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.10.40.125:46051: Network error: Client connection negotiation failed: client connection to 127.10.40.125:46051: connect: Connection refused (error 111)
08:16:17.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.136039 10500 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.40.125" port: 46051 } attempt: 2
08:16:17.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:17.138533 10500 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.10.40.125:46051: Network error: Client connection negotiation failed: client connection to 127.10.40.125:46051: connect: Connection refused (error 111)
08:16:17.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.229851 10500 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.40.125" port: 46051 } attempt: 3
08:16:17.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:17.232676 10500 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.10.40.125:46051: Network error: Client connection negotiation failed: client connection to 127.10.40.125:46051: connect: Connection refused (error 111)
08:16:17.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:17.366326 10503 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:16:17.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:17.366897 10503 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:16:17.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:17.367020 10503 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:16:17.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.375975 10500 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.40.125" port: 46051 } attempt: 4
08:16:17.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:17.376310 10503 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
08:16:17.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:17.376410 10503 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:16:17.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:17.376507 10503 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
08:16:17.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:17.376579 10503 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
08:16:17.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:17.376672 10503 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
08:16:17.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:17.376729 10503 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:34289/jwks.json
08:16:17.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:17.379437 10500 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.10.40.125:46051: Network error: Client connection negotiation failed: client connection to 127.10.40.125:46051: connect: Connection refused (error 111)
08:16:17.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.388371 10503 master_runner.cc:387] Master server non-default flags:
08:16:17.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-1/logs
08:16:17.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:16:17.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:16:17.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:16:17.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.40.84:40955
08:16:17.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:16:17.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:16:17.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-1/data
08:16:17.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-1/wal
08:16:17.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
08:16:17.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.10.40.126:45793,127.10.40.125:46051,127.10.40.124:41773
08:16:17.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:16:17.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:16:17.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
08:16:17.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.40.125:46051
08:16:17.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
08:16:17.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:16:17.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
08:16:17.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34289/jwks.json
08:16:17.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:16:17.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:16:17.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-1/data/info.pb
08:16:17.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.40.125
08:16:17.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
08:16:17.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:16:17.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:16:17.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:16:17.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:16:17.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:16:17.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:16:17.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
08:16:17.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:16:17.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 70c51312f1186a8ae9f46647160b7730123f7601
08:16:17.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:16:17.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 12 Aug 2026 07:43:21 UTC on bdcb31816ec0
08:16:17.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 13302
08:16:17.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
08:16:17.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.390837 10503 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:16:17.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.393965 10503 file_cache.cc:504] Constructed file cache file cache with capacity 419430
08:16:17.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:17.405709 10512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:16:17.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:17.408560 10513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:16:17.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.412607 10503 server_base.cc:1061] running on GCE node
08:16:17.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:17.413359 10516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:16:17.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.415156 10503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:16:17.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.417363 10503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:16:17.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.418737 10503 hybrid_clock.cc:648] HybridClock initialized: now 1786522577418544 us; error 66 us; skew 500 ppm
08:16:17.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.419431 10503 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:16:17.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.423454 10503 webserver.cc:533] Webserver started at http://127.10.40.125:43479/ using document root <none> and password file <none>
08:16:17.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.424530 10503 fs_manager.cc:362] Metadata directory not provided
08:16:17.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.424827 10503 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:16:17.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.425603 10503 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
08:16:17.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.428483 10503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-1/data/instance:
08:16:17.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b7c1f40057e3404093b0ed154254604f"
08:16:17.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:16:17 on dist-test-slave-gp6n"
08:16:17.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.429519 10503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-1/wal/instance:
08:16:17.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b7c1f40057e3404093b0ed154254604f"
08:16:17.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:16:17 on dist-test-slave-gp6n"
08:16:17.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.435815 10503 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.006s	sys 0.001s
08:16:17.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.440192 10523 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:16:17.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.442080 10503 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.000s
08:16:17.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.442342 10503 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-1/data,/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-1/wal
08:16:17.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b7c1f40057e3404093b0ed154254604f"
08:16:17.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:16:17 on dist-test-slave-gp6n"
08:16:17.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.442682 10503 fs_report.cc:389] FS layout report
08:16:17.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:16:17.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-1/wal
08:16:17.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-1/wal
08:16:17.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-1/data/data
08:16:17.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:16:17.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:16:17.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:16:17.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:16:17.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:16:17.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:16:17.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:16:17.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:16:17.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:16:17.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:16:17.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:16:17.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.469722 10503 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:16:17.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.471191 10503 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:16:17.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.471642 10503 kserver.cc:163] Server-wide thread pool size limit: 3276
08:16:17.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.496675 10503 rpc_server.cc:307] RPC server started. Bound to: 127.10.40.125:46051
08:16:17.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.496706 10597 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.40.125:46051 every 8 connection(s)
08:16:17.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.498811 10503 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-1/data/info.pb
08:16:17.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.503719 10598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
08:16:17.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.507851 10401 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0lHOBl/build/asan/bin/kudu as pid 10503
08:16:17.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.508049 10401 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-1/wal/instance
08:16:17.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.509402 10401 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0lHOBl/build/asan/bin/kudu
08:16:17.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task0lHOBl/build/asan/bin/kudu
08:16:17.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-2/wal
08:16:17.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-2/data
08:16:17.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:16:17.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:16:17.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:16:17.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:16:17.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:16:17.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:16:17.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-2/logs
08:16:17.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-2/data/info.pb
08:16:17.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:16:17.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:16:17.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:16:17.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:16:17.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:16:17.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:16:17.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:16:17.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:16:17.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
08:16:17.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:16:17.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
08:16:17.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
08:16:17.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.40.124:41773
08:16:17.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.40.124
08:16:17.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
08:16:17.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.40.84:40955
08:16:17.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:16:17.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:16:17.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:16:17.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
08:16:17.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.10.40.126:45793,127.10.40.125:46051,127.10.40.124:41773
08:16:17.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
08:16:17.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34289/jwks.json with env {}
08:16:17.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.513481 10598 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.40.126" port: 45793 } has no permanent_uuid. Determining permanent_uuid...
08:16:17.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.533632 10598 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.40.125" port: 46051 } has no permanent_uuid. Determining permanent_uuid...
08:16:17.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.543797 10598 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.40.124" port: 41773 } has no permanent_uuid. Determining permanent_uuid...
08:16:17.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:17.545117 10527 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.40.124:41773: connect: Connection refused (error 111)
08:16:17.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:17.547098 10598 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.10.40.124:41773: Network error: Client connection negotiation failed: client connection to 127.10.40.124:41773: connect: Connection refused (error 111)
08:16:17.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.596452 10598 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.40.124" port: 41773 } attempt: 1
08:16:17.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:17.598747 10598 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.10.40.124:41773: Network error: Client connection negotiation failed: client connection to 127.10.40.124:41773: connect: Connection refused (error 111)
08:16:17.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.667173 10598 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.40.124" port: 41773 } attempt: 2
08:16:17.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:17.669673 10598 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.10.40.124:41773: Network error: Client connection negotiation failed: client connection to 127.10.40.124:41773: connect: Connection refused (error 111)
08:16:17.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.678695 10500 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.40.125" port: 46051 } attempt: 5
08:16:17.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.686110 10500 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.40.124" port: 41773 } has no permanent_uuid. Determining permanent_uuid...
08:16:17.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:17.688271 10500 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.10.40.124:41773: Network error: Client connection negotiation failed: client connection to 127.10.40.124:41773: connect: Connection refused (error 111)
08:16:17.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.739521 10500 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.40.124" port: 41773 } attempt: 1
08:16:17.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:17.741729 10500 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.10.40.124:41773: Network error: Client connection negotiation failed: client connection to 127.10.40.124:41773: connect: Connection refused (error 111)
08:16:17.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.760932 10598 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.40.124" port: 41773 } attempt: 3
08:16:17.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:17.763778 10598 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.10.40.124:41773: Network error: Client connection negotiation failed: client connection to 127.10.40.124:41773: connect: Connection refused (error 111)
08:16:17.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.809964 10500 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.40.124" port: 41773 } attempt: 2
08:16:17.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:17.812490 10500 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.10.40.124:41773: Network error: Client connection negotiation failed: client connection to 127.10.40.124:41773: connect: Connection refused (error 111)
08:16:17.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:17.886096 10603 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:16:17.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:17.886490 10603 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:16:17.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:17.886582 10603 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:16:17.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:17.895051 10603 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
08:16:17.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:17.895134 10603 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:16:17.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:17.895191 10603 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
08:16:17.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:17.895232 10603 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
08:16:17.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:17.895344 10603 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
08:16:17.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:17.895383 10603 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:34289/jwks.json
08:16:17.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.905498 10603 master_runner.cc:387] Master server non-default flags:
08:16:17.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-2/logs
08:16:17.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:16:17.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:16:17.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:16:17.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.40.84:40955
08:16:17.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:16:17.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:16:17.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-2/data
08:16:17.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-2/wal
08:16:17.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
08:16:17.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.10.40.126:45793,127.10.40.125:46051,127.10.40.124:41773
08:16:17.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:16:17.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:16:17.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
08:16:17.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.40.124:41773
08:16:17.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
08:16:17.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:16:17.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
08:16:17.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34289/jwks.json
08:16:17.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:16:17.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:16:17.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-2/data/info.pb
08:16:17.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.40.124
08:16:17.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
08:16:17.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:16:17.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:16:17.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:16:17.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:16:17.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:16:17.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:16:17.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
08:16:17.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:16:17.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 70c51312f1186a8ae9f46647160b7730123f7601
08:16:17.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:16:17.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 12 Aug 2026 07:43:21 UTC on bdcb31816ec0
08:16:17.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 13302
08:16:17.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
08:16:17.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.907059 10598 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.40.124" port: 41773 } attempt: 4
08:16:17.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.907473 10603 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:16:17.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:17.909621 10598 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.10.40.124:41773: Network error: Client connection negotiation failed: client connection to 127.10.40.124:41773: connect: Connection refused (error 111)
08:16:17.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.910020 10603 file_cache.cc:504] Constructed file cache file cache with capacity 419430
08:16:17.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.918746 10500 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.40.124" port: 41773 } attempt: 3
08:16:17.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:17.918769 10614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:16:17.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:17.919456 10615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:16:17.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:17.919581 10617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:16:17.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:17.920830 10500 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.10.40.124:41773: Network error: Client connection negotiation failed: client connection to 127.10.40.124:41773: connect: Connection refused (error 111)
08:16:17.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.921149 10603 server_base.cc:1061] running on GCE node
08:16:17.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.921844 10603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:16:17.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.923327 10603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:16:17.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.924572 10603 hybrid_clock.cc:648] HybridClock initialized: now 1786522577924513 us; error 43 us; skew 500 ppm
08:16:17.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.925047 10603 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:16:17.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.927440 10603 webserver.cc:533] Webserver started at http://127.10.40.124:41359/ using document root <none> and password file <none>
08:16:17.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.928262 10603 fs_manager.cc:362] Metadata directory not provided
08:16:17.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.928395 10603 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:16:17.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.928972 10603 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
08:16:17.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.931645 10603 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-2/data/instance:
08:16:17.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "88955d5e7d8b4dc8a58cf089ab03559c"
08:16:17.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:16:17 on dist-test-slave-gp6n"
08:16:17.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.932616 10603 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-2/wal/instance:
08:16:17.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "88955d5e7d8b4dc8a58cf089ab03559c"
08:16:17.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:16:17 on dist-test-slave-gp6n"
08:16:17.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.938254 10603 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.002s	sys 0.005s
08:16:17.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.941857 10627 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:16:17.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.943300 10603 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
08:16:17.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.943508 10603 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-2/data,/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-2/wal
08:16:17.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "88955d5e7d8b4dc8a58cf089ab03559c"
08:16:17.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:16:17 on dist-test-slave-gp6n"
08:16:17.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.943782 10603 fs_report.cc:389] FS layout report
08:16:17.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:16:17.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-2/wal
08:16:17.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-2/wal
08:16:17.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-2/data/data
08:16:17.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:16:17.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:16:17.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:16:17.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:16:17.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:16:17.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:16:17.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:16:17.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:16:17.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:16:17.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:16:17.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:16:17.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.964596 10603 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:16:17.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.965689 10603 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:16:17.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.966063 10603 kserver.cc:163] Server-wide thread pool size limit: 3276
08:16:17.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.988651 10603 rpc_server.cc:307] RPC server started. Bound to: 127.10.40.124:41773
08:16:17.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.988691 10689 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.40.124:41773 every 8 connection(s)
08:16:17.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.990525 10603 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-2/data/info.pb
08:16:17.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.994890 10690 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
08:16:17.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.998433 10401 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0lHOBl/build/asan/bin/kudu as pid 10603
08:16:17.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:17.998652 10401 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/master-2/wal/instance
08:16:18.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.002379 10690 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.40.126" port: 45793 } has no permanent_uuid. Determining permanent_uuid...
08:16:18.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.031229 10690 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.40.125" port: 46051 } has no permanent_uuid. Determining permanent_uuid...
08:16:18.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.038151 10690 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.40.124" port: 41773 } has no permanent_uuid. Determining permanent_uuid...
08:16:18.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.049528 10690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88955d5e7d8b4dc8a58cf089ab03559c: Bootstrap starting.
08:16:18.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.053134 10690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88955d5e7d8b4dc8a58cf089ab03559c: Neither blocks nor log segments found. Creating new log.
08:16:18.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.054365 10690 log.cc:826] T 00000000000000000000000000000000 P 88955d5e7d8b4dc8a58cf089ab03559c: Log is configured to *not* fsync() on all Append() calls
08:16:18.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.057523 10690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88955d5e7d8b4dc8a58cf089ab03559c: No bootstrap required, opened a new log
08:16:18.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.063450 10690 raft_consensus.cc:359] T 00000000000000000000000000000000 P 88955d5e7d8b4dc8a58cf089ab03559c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b587403b6a4d3090cfced49037efc0" member_type: VOTER last_known_addr { host: "127.10.40.126" port: 45793 } } peers { permanent_uuid: "b7c1f40057e3404093b0ed154254604f" member_type: VOTER last_known_addr { host: "127.10.40.125" port: 46051 } } peers { permanent_uuid: "88955d5e7d8b4dc8a58cf089ab03559c" member_type: VOTER last_known_addr { host: "127.10.40.124" port: 41773 } }
08:16:18.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.063759 10690 raft_consensus.cc:385] T 00000000000000000000000000000000 P 88955d5e7d8b4dc8a58cf089ab03559c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
08:16:18.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.063885 10690 raft_consensus.cc:740] T 00000000000000000000000000000000 P 88955d5e7d8b4dc8a58cf089ab03559c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88955d5e7d8b4dc8a58cf089ab03559c, State: Initialized, Role: FOLLOWER
08:16:18.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.064626 10690 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88955d5e7d8b4dc8a58cf089ab03559c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b587403b6a4d3090cfced49037efc0" member_type: VOTER last_known_addr { host: "127.10.40.126" port: 45793 } } peers { permanent_uuid: "b7c1f40057e3404093b0ed154254604f" member_type: VOTER last_known_addr { host: "127.10.40.125" port: 46051 } } peers { permanent_uuid: "88955d5e7d8b4dc8a58cf089ab03559c" member_type: VOTER last_known_addr { host: "127.10.40.124" port: 41773 } }
08:16:18.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.065598 10701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88955d5e7d8b4dc8a58cf089ab03559c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b587403b6a4d3090cfced49037efc0" member_type: VOTER last_known_addr { host: "127.10.40.126" port: 45793 } } peers { permanent_uuid: "b7c1f40057e3404093b0ed154254604f" member_type: VOTER last_known_addr { host: "127.10.40.125" port: 46051 } } peers { permanent_uuid: "88955d5e7d8b4dc8a58cf089ab03559c" member_type: VOTER last_known_addr { host: "127.10.40.124" port: 41773 } } }
08:16:18.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.065878 10701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88955d5e7d8b4dc8a58cf089ab03559c [sys.catalog]: This master's current role is: FOLLOWER
08:16:18.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.066882 10690 sys_catalog.cc:565] T 00000000000000000000000000000000 P 88955d5e7d8b4dc8a58cf089ab03559c [sys.catalog]: configured and running, proceeding with master startup.
08:16:18.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:18.078676 10716 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 88955d5e7d8b4dc8a58cf089ab03559c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
08:16:18.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:18.079305 10716 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
08:16:18.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.075809 10603 master_runner.cc:187] Error getting master registration for 127.10.40.126:45793: OK, instance_id { permanent_uuid: "a0b587403b6a4d3090cfced49037efc0" instance_seqno: 1786522576974356 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
08:16:18.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.081923 10603 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
08:16:18.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.098085 10500 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.40.124" port: 41773 } attempt: 4
08:16:18.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.109050 10500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0b587403b6a4d3090cfced49037efc0: Bootstrap starting.
08:16:18.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.112504 10500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0b587403b6a4d3090cfced49037efc0: Neither blocks nor log segments found. Creating new log.
08:16:18.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.113814 10500 log.cc:826] T 00000000000000000000000000000000 P a0b587403b6a4d3090cfced49037efc0: Log is configured to *not* fsync() on all Append() calls
08:16:18.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.117170 10500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0b587403b6a4d3090cfced49037efc0: No bootstrap required, opened a new log
08:16:18.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.123148 10500 raft_consensus.cc:359] T 00000000000000000000000000000000 P a0b587403b6a4d3090cfced49037efc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b587403b6a4d3090cfced49037efc0" member_type: VOTER last_known_addr { host: "127.10.40.126" port: 45793 } } peers { permanent_uuid: "b7c1f40057e3404093b0ed154254604f" member_type: VOTER last_known_addr { host: "127.10.40.125" port: 46051 } } peers { permanent_uuid: "88955d5e7d8b4dc8a58cf089ab03559c" member_type: VOTER last_known_addr { host: "127.10.40.124" port: 41773 } }
08:16:18.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.123457 10500 raft_consensus.cc:385] T 00000000000000000000000000000000 P a0b587403b6a4d3090cfced49037efc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
08:16:18.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.123590 10500 raft_consensus.cc:740] T 00000000000000000000000000000000 P a0b587403b6a4d3090cfced49037efc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0b587403b6a4d3090cfced49037efc0, State: Initialized, Role: FOLLOWER
08:16:18.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.124389 10500 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0b587403b6a4d3090cfced49037efc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b587403b6a4d3090cfced49037efc0" member_type: VOTER last_known_addr { host: "127.10.40.126" port: 45793 } } peers { permanent_uuid: "b7c1f40057e3404093b0ed154254604f" member_type: VOTER last_known_addr { host: "127.10.40.125" port: 46051 } } peers { permanent_uuid: "88955d5e7d8b4dc8a58cf089ab03559c" member_type: VOTER last_known_addr { host: "127.10.40.124" port: 41773 } }
08:16:18.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.125454 10719 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0b587403b6a4d3090cfced49037efc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b587403b6a4d3090cfced49037efc0" member_type: VOTER last_known_addr { host: "127.10.40.126" port: 45793 } } peers { permanent_uuid: "b7c1f40057e3404093b0ed154254604f" member_type: VOTER last_known_addr { host: "127.10.40.125" port: 46051 } } peers { permanent_uuid: "88955d5e7d8b4dc8a58cf089ab03559c" member_type: VOTER last_known_addr { host: "127.10.40.124" port: 41773 } } }
08:16:18.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.125736 10719 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0b587403b6a4d3090cfced49037efc0 [sys.catalog]: This master's current role is: FOLLOWER
08:16:18.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.126669 10500 sys_catalog.cc:565] T 00000000000000000000000000000000 P a0b587403b6a4d3090cfced49037efc0 [sys.catalog]: configured and running, proceeding with master startup.
08:16:18.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:18.139101 10733 catalog_manager.cc:1594] T 00000000000000000000000000000000 P a0b587403b6a4d3090cfced49037efc0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
08:16:18.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:18.139742 10733 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
08:16:18.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.138492 10414 master_runner.cc:187] Error getting master registration for 127.10.40.125:46051: OK, instance_id { permanent_uuid: "b7c1f40057e3404093b0ed154254604f" instance_seqno: 1786522577489646 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
08:16:18.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.144076 10414 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
08:16:18.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.208882 10598 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.40.124" port: 41773 } attempt: 5
08:16:18.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:18.212116 10437 tablet.cc:2406] T 00000000000000000000000000000000 P a0b587403b6a4d3090cfced49037efc0: Can't schedule compaction. Clean time has not been advanced past its initial value.
08:16:18.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:18.218510 10635 tablet.cc:2406] T 00000000000000000000000000000000 P 88955d5e7d8b4dc8a58cf089ab03559c: Can't schedule compaction. Clean time has not been advanced past its initial value.
08:16:18.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.222522 10598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7c1f40057e3404093b0ed154254604f: Bootstrap starting.
08:16:18.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.226923 10598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7c1f40057e3404093b0ed154254604f: Neither blocks nor log segments found. Creating new log.
08:16:18.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.229192 10598 log.cc:826] T 00000000000000000000000000000000 P b7c1f40057e3404093b0ed154254604f: Log is configured to *not* fsync() on all Append() calls
08:16:18.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.234069 10598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7c1f40057e3404093b0ed154254604f: No bootstrap required, opened a new log
08:16:18.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.242266 10598 raft_consensus.cc:359] T 00000000000000000000000000000000 P b7c1f40057e3404093b0ed154254604f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b587403b6a4d3090cfced49037efc0" member_type: VOTER last_known_addr { host: "127.10.40.126" port: 45793 } } peers { permanent_uuid: "b7c1f40057e3404093b0ed154254604f" member_type: VOTER last_known_addr { host: "127.10.40.125" port: 46051 } } peers { permanent_uuid: "88955d5e7d8b4dc8a58cf089ab03559c" member_type: VOTER last_known_addr { host: "127.10.40.124" port: 41773 } }
08:16:18.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.242690 10598 raft_consensus.cc:385] T 00000000000000000000000000000000 P b7c1f40057e3404093b0ed154254604f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
08:16:18.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.242843 10598 raft_consensus.cc:740] T 00000000000000000000000000000000 P b7c1f40057e3404093b0ed154254604f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7c1f40057e3404093b0ed154254604f, State: Initialized, Role: FOLLOWER
08:16:18.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.244016 10598 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7c1f40057e3404093b0ed154254604f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b587403b6a4d3090cfced49037efc0" member_type: VOTER last_known_addr { host: "127.10.40.126" port: 45793 } } peers { permanent_uuid: "b7c1f40057e3404093b0ed154254604f" member_type: VOTER last_known_addr { host: "127.10.40.125" port: 46051 } } peers { permanent_uuid: "88955d5e7d8b4dc8a58cf089ab03559c" member_type: VOTER last_known_addr { host: "127.10.40.124" port: 41773 } }
08:16:18.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.251534 10736 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7c1f40057e3404093b0ed154254604f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b587403b6a4d3090cfced49037efc0" member_type: VOTER last_known_addr { host: "127.10.40.126" port: 45793 } } peers { permanent_uuid: "b7c1f40057e3404093b0ed154254604f" member_type: VOTER last_known_addr { host: "127.10.40.125" port: 46051 } } peers { permanent_uuid: "88955d5e7d8b4dc8a58cf089ab03559c" member_type: VOTER last_known_addr { host: "127.10.40.124" port: 41773 } } }
08:16:18.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.254128 10736 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7c1f40057e3404093b0ed154254604f [sys.catalog]: This master's current role is: FOLLOWER
08:16:18.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.253991 10598 sys_catalog.cc:565] T 00000000000000000000000000000000 P b7c1f40057e3404093b0ed154254604f [sys.catalog]: configured and running, proceeding with master startup.
08:16:18.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:18.276506 10755 catalog_manager.cc:1594] T 00000000000000000000000000000000 P b7c1f40057e3404093b0ed154254604f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
08:16:18.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:18.277293 10755 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
08:16:18.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.280413 10503 master_runner.cc:264] No leader master found from master b7c1f40057e3404093b0ed154254604f
08:16:18.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.280530 10503 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
08:16:18.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.310186 10701 raft_consensus.cc:493] T 00000000000000000000000000000000 P 88955d5e7d8b4dc8a58cf089ab03559c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
08:16:18.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.310603 10701 raft_consensus.cc:515] T 00000000000000000000000000000000 P 88955d5e7d8b4dc8a58cf089ab03559c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b587403b6a4d3090cfced49037efc0" member_type: VOTER last_known_addr { host: "127.10.40.126" port: 45793 } } peers { permanent_uuid: "b7c1f40057e3404093b0ed154254604f" member_type: VOTER last_known_addr { host: "127.10.40.125" port: 46051 } } peers { permanent_uuid: "88955d5e7d8b4dc8a58cf089ab03559c" member_type: VOTER last_known_addr { host: "127.10.40.124" port: 41773 } }
08:16:18.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.312580 10701 leader_election.cc:290] T 00000000000000000000000000000000 P 88955d5e7d8b4dc8a58cf089ab03559c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0b587403b6a4d3090cfced49037efc0 (127.10.40.126:45793), b7c1f40057e3404093b0ed154254604f (127.10.40.125:46051)
08:16:18.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.315485 10562 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88955d5e7d8b4dc8a58cf089ab03559c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7c1f40057e3404093b0ed154254604f" is_pre_election: true
08:16:18.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.316144 10562 raft_consensus.cc:2468] T 00000000000000000000000000000000 P b7c1f40057e3404093b0ed154254604f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88955d5e7d8b4dc8a58cf089ab03559c in term 0.
08:16:18.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.316946 10468 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88955d5e7d8b4dc8a58cf089ab03559c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0b587403b6a4d3090cfced49037efc0" is_pre_election: true
08:16:18.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.317469 10468 raft_consensus.cc:2468] T 00000000000000000000000000000000 P a0b587403b6a4d3090cfced49037efc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88955d5e7d8b4dc8a58cf089ab03559c in term 0.
08:16:18.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.318334 10630 leader_election.cc:304] T 00000000000000000000000000000000 P 88955d5e7d8b4dc8a58cf089ab03559c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88955d5e7d8b4dc8a58cf089ab03559c, a0b587403b6a4d3090cfced49037efc0; no voters: 
08:16:18.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.325387 10701 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 88955d5e7d8b4dc8a58cf089ab03559c [term 0 FOLLOWER]: Leader pre-election won for term 1
08:16:18.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.325621 10701 raft_consensus.cc:493] T 00000000000000000000000000000000 P 88955d5e7d8b4dc8a58cf089ab03559c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
08:16:18.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.325738 10701 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 88955d5e7d8b4dc8a58cf089ab03559c [term 0 FOLLOWER]: Advancing to term 1
08:16:18.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.330471 10701 raft_consensus.cc:515] T 00000000000000000000000000000000 P 88955d5e7d8b4dc8a58cf089ab03559c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b587403b6a4d3090cfced49037efc0" member_type: VOTER last_known_addr { host: "127.10.40.126" port: 45793 } } peers { permanent_uuid: "b7c1f40057e3404093b0ed154254604f" member_type: VOTER last_known_addr { host: "127.10.40.125" port: 46051 } } peers { permanent_uuid: "88955d5e7d8b4dc8a58cf089ab03559c" member_type: VOTER last_known_addr { host: "127.10.40.124" port: 41773 } }
08:16:18.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.331848 10468 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88955d5e7d8b4dc8a58cf089ab03559c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0b587403b6a4d3090cfced49037efc0"
08:16:18.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.332113 10468 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a0b587403b6a4d3090cfced49037efc0 [term 0 FOLLOWER]: Advancing to term 1
08:16:18.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.337124 10468 raft_consensus.cc:2468] T 00000000000000000000000000000000 P a0b587403b6a4d3090cfced49037efc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88955d5e7d8b4dc8a58cf089ab03559c in term 1.
08:16:18.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.337733 10630 leader_election.cc:304] T 00000000000000000000000000000000 P 88955d5e7d8b4dc8a58cf089ab03559c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88955d5e7d8b4dc8a58cf089ab03559c, a0b587403b6a4d3090cfced49037efc0; no voters: 
08:16:18.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.340314 10701 leader_election.cc:290] T 00000000000000000000000000000000 P 88955d5e7d8b4dc8a58cf089ab03559c [CANDIDATE]: Term 1 election: Requested vote from peers a0b587403b6a4d3090cfced49037efc0 (127.10.40.126:45793), b7c1f40057e3404093b0ed154254604f (127.10.40.125:46051)
08:16:18.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.341039 10701 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 88955d5e7d8b4dc8a58cf089ab03559c [term 1 FOLLOWER]: Leader election won for term 1
08:16:18.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.341202 10701 raft_consensus.cc:697] T 00000000000000000000000000000000 P 88955d5e7d8b4dc8a58cf089ab03559c [term 1 LEADER]: Becoming Leader. State: Replica: 88955d5e7d8b4dc8a58cf089ab03559c, State: Running, Role: LEADER
08:16:18.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.341773 10701 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88955d5e7d8b4dc8a58cf089ab03559c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b587403b6a4d3090cfced49037efc0" member_type: VOTER last_known_addr { host: "127.10.40.126" port: 45793 } } peers { permanent_uuid: "b7c1f40057e3404093b0ed154254604f" member_type: VOTER last_known_addr { host: "127.10.40.125" port: 46051 } } peers { permanent_uuid: "88955d5e7d8b4dc8a58cf089ab03559c" member_type: VOTER last_known_addr { host: "127.10.40.124" port: 41773 } }
08:16:18.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.356654 10562 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88955d5e7d8b4dc8a58cf089ab03559c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7c1f40057e3404093b0ed154254604f"
08:16:18.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.356922 10562 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b7c1f40057e3404093b0ed154254604f [term 0 FOLLOWER]: Advancing to term 1
08:16:18.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.360742 10562 raft_consensus.cc:2468] T 00000000000000000000000000000000 P b7c1f40057e3404093b0ed154254604f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88955d5e7d8b4dc8a58cf089ab03559c in term 1.
08:16:18.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.365386 10757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88955d5e7d8b4dc8a58cf089ab03559c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88955d5e7d8b4dc8a58cf089ab03559c. Latest consensus state: current_term: 1 leader_uuid: "88955d5e7d8b4dc8a58cf089ab03559c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b587403b6a4d3090cfced49037efc0" member_type: VOTER last_known_addr { host: "127.10.40.126" port: 45793 } } peers { permanent_uuid: "b7c1f40057e3404093b0ed154254604f" member_type: VOTER last_known_addr { host: "127.10.40.125" port: 46051 } } peers { permanent_uuid: "88955d5e7d8b4dc8a58cf089ab03559c" member_type: VOTER last_known_addr { host: "127.10.40.124" port: 41773 } } }
08:16:18.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.366101 10757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88955d5e7d8b4dc8a58cf089ab03559c [sys.catalog]: This master's current role is: LEADER
08:16:18.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.373509 10761 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
08:16:18.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.376600 10761 catalog_manager.cc:1520] Initializing Kudu cluster ID...
08:16:18.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.397480 10562 raft_consensus.cc:1275] T 00000000000000000000000000000000 P b7c1f40057e3404093b0ed154254604f [term 1 FOLLOWER]: Refusing update from remote peer 88955d5e7d8b4dc8a58cf089ab03559c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
08:16:18.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.399296 10757 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 88955d5e7d8b4dc8a58cf089ab03559c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7c1f40057e3404093b0ed154254604f" member_type: VOTER last_known_addr { host: "127.10.40.125" port: 46051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
08:16:18.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.407464 10468 raft_consensus.cc:1275] T 00000000000000000000000000000000 P a0b587403b6a4d3090cfced49037efc0 [term 1 FOLLOWER]: Refusing update from remote peer 88955d5e7d8b4dc8a58cf089ab03559c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
08:16:18.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.409230 10757 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 88955d5e7d8b4dc8a58cf089ab03559c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0b587403b6a4d3090cfced49037efc0" member_type: VOTER last_known_addr { host: "127.10.40.126" port: 45793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
08:16:18.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.432905 10719 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0b587403b6a4d3090cfced49037efc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88955d5e7d8b4dc8a58cf089ab03559c. Latest consensus state: current_term: 1 leader_uuid: "88955d5e7d8b4dc8a58cf089ab03559c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b587403b6a4d3090cfced49037efc0" member_type: VOTER last_known_addr { host: "127.10.40.126" port: 45793 } } peers { permanent_uuid: "b7c1f40057e3404093b0ed154254604f" member_type: VOTER last_known_addr { host: "127.10.40.125" port: 46051 } } peers { permanent_uuid: "88955d5e7d8b4dc8a58cf089ab03559c" member_type: VOTER last_known_addr { host: "127.10.40.124" port: 41773 } } }
08:16:18.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.433215 10719 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0b587403b6a4d3090cfced49037efc0 [sys.catalog]: This master's current role is: FOLLOWER
08:16:18.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.439316 10762 mvcc.cc:204] Tried to move back new op lower bound from 7317596481104130048 to 7317596480924200960. Current Snapshot: MvccSnapshot[applied={T|T < 7317596481104130048}]
08:16:18.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.448945 10701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88955d5e7d8b4dc8a58cf089ab03559c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "88955d5e7d8b4dc8a58cf089ab03559c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b587403b6a4d3090cfced49037efc0" member_type: VOTER last_known_addr { host: "127.10.40.126" port: 45793 } } peers { permanent_uuid: "b7c1f40057e3404093b0ed154254604f" member_type: VOTER last_known_addr { host: "127.10.40.125" port: 46051 } } peers { permanent_uuid: "88955d5e7d8b4dc8a58cf089ab03559c" member_type: VOTER last_known_addr { host: "127.10.40.124" port: 41773 } } }
08:16:18.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.449312 10701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88955d5e7d8b4dc8a58cf089ab03559c [sys.catalog]: This master's current role is: LEADER
08:16:18.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.451647 10761 catalog_manager.cc:1383] Generated new cluster ID: db600596164f4994a1198e869813dd76
08:16:18.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.451771 10761 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
08:16:18.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.460664 10719 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0b587403b6a4d3090cfced49037efc0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "88955d5e7d8b4dc8a58cf089ab03559c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b587403b6a4d3090cfced49037efc0" member_type: VOTER last_known_addr { host: "127.10.40.126" port: 45793 } } peers { permanent_uuid: "b7c1f40057e3404093b0ed154254604f" member_type: VOTER last_known_addr { host: "127.10.40.125" port: 46051 } } peers { permanent_uuid: "88955d5e7d8b4dc8a58cf089ab03559c" member_type: VOTER last_known_addr { host: "127.10.40.124" port: 41773 } } }
08:16:18.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.460928 10719 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0b587403b6a4d3090cfced49037efc0 [sys.catalog]: This master's current role is: FOLLOWER
08:16:18.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.472800 10736 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7c1f40057e3404093b0ed154254604f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88955d5e7d8b4dc8a58cf089ab03559c. Latest consensus state: current_term: 1 leader_uuid: "88955d5e7d8b4dc8a58cf089ab03559c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b587403b6a4d3090cfced49037efc0" member_type: VOTER last_known_addr { host: "127.10.40.126" port: 45793 } } peers { permanent_uuid: "b7c1f40057e3404093b0ed154254604f" member_type: VOTER last_known_addr { host: "127.10.40.125" port: 46051 } } peers { permanent_uuid: "88955d5e7d8b4dc8a58cf089ab03559c" member_type: VOTER last_known_addr { host: "127.10.40.124" port: 41773 } } }
08:16:18.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.473066 10736 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7c1f40057e3404093b0ed154254604f [sys.catalog]: This master's current role is: FOLLOWER
08:16:18.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:18.476296 10531 tablet.cc:2406] T 00000000000000000000000000000000 P b7c1f40057e3404093b0ed154254604f: Can't schedule compaction. Clean time has not been advanced past its initial value.
08:16:18.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.476432 10757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88955d5e7d8b4dc8a58cf089ab03559c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "88955d5e7d8b4dc8a58cf089ab03559c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b587403b6a4d3090cfced49037efc0" member_type: VOTER last_known_addr { host: "127.10.40.126" port: 45793 } } peers { permanent_uuid: "b7c1f40057e3404093b0ed154254604f" member_type: VOTER last_known_addr { host: "127.10.40.125" port: 46051 } } peers { permanent_uuid: "88955d5e7d8b4dc8a58cf089ab03559c" member_type: VOTER last_known_addr { host: "127.10.40.124" port: 41773 } } }
08:16:18.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.476636 10757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88955d5e7d8b4dc8a58cf089ab03559c [sys.catalog]: This master's current role is: LEADER
08:16:18.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.478614 10764 mvcc.cc:204] Tried to move back new op lower bound from 7317596481104130048 to 7317596480924200960. Current Snapshot: MvccSnapshot[applied={T|T < 7317596481104130048}]
08:16:18.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.488986 10736 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7c1f40057e3404093b0ed154254604f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "88955d5e7d8b4dc8a58cf089ab03559c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b587403b6a4d3090cfced49037efc0" member_type: VOTER last_known_addr { host: "127.10.40.126" port: 45793 } } peers { permanent_uuid: "b7c1f40057e3404093b0ed154254604f" member_type: VOTER last_known_addr { host: "127.10.40.125" port: 46051 } } peers { permanent_uuid: "88955d5e7d8b4dc8a58cf089ab03559c" member_type: VOTER last_known_addr { host: "127.10.40.124" port: 41773 } } }
08:16:18.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.489270 10736 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7c1f40057e3404093b0ed154254604f [sys.catalog]: This master's current role is: FOLLOWER
08:16:18.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.512957 10761 catalog_manager.cc:1406] Generated new certificate authority record
08:16:18.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.515882 10761 catalog_manager.cc:1540] Loading token signing keys...
08:16:18.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.540658 10761 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 88955d5e7d8b4dc8a58cf089ab03559c: Generated new TSK 0
08:16:18.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.543766 10761 catalog_manager.cc:1550] Initializing in-progress tserver states...
08:16:18.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.574651 10401 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0lHOBl/build/asan/bin/kudu
08:16:18.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task0lHOBl/build/asan/bin/kudu
08:16:18.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/wal
08:16:18.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/data
08:16:18.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:16:18.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:16:18.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:16:18.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:16:18.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:16:18.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:16:18.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/logs
08:16:18.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/data/info.pb
08:16:18.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:16:18.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:16:18.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:16:18.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:16:18.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:16:18.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:16:18.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:16:18.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:16:18.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:16:18.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:16:18.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.40.65:0
08:16:18.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.40.65
08:16:18.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.40.65
08:16:18.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
08:16:18.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.40.126:45793,127.10.40.125:46051,127.10.40.124:41773
08:16:18.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.40.84:40955
08:16:18.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:16:18.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:16:18.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:16:18.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:18.969805 10771 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:16:18.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:18.970136 10771 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:16:18.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:18.970266 10771 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:16:18.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:18.979220 10771 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:16:18.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:18.979396 10771 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.40.65
08:16:18.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.996186 10771 tablet_server_runner.cc:78] Tablet server non-default flags:
08:16:18.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/logs
08:16:18.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:16:18.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:16:18.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:16:18.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.40.84:40955
08:16:18.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:16:18.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:16:18.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/data
08:16:18.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/wal
08:16:18.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:16:18.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:16:18.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.40.65:0
08:16:18.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:16:18.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:16:18.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:16:18.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/data/info.pb
08:16:18.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.40.65
08:16:18.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
08:16:18.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.40.126:45793,127.10.40.125:46051,127.10.40.124:41773
08:16:18.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:16:18.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:16:18.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:16:18.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:16:18.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:16:18.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.40.65
08:16:18.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:16:18.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:16:18.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:16:19.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 70c51312f1186a8ae9f46647160b7730123f7601
08:16:19.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:16:19.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 12 Aug 2026 07:43:21 UTC on bdcb31816ec0
08:16:19.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 13302
08:16:19.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
08:16:19.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:18.998284 10771 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:16:19.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.000622 10771 file_cache.cc:504] Constructed file cache file cache with capacity 419430
08:16:19.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:19.012028 10777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:16:19.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:19.013643 10779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:16:19.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:19.019896 10781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:16:19.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.020398 10771 server_base.cc:1061] running on GCE node
08:16:19.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.021068 10771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:16:19.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.022265 10771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:16:19.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.023530 10771 hybrid_clock.cc:648] HybridClock initialized: now 1786522579023466 us; error 52 us; skew 500 ppm
08:16:19.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.023926 10771 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:16:19.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.026512 10771 webserver.cc:533] Webserver started at http://127.10.40.65:41939/ using document root <none> and password file <none>
08:16:19.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.027383 10771 fs_manager.cc:362] Metadata directory not provided
08:16:19.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.027521 10771 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:16:19.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.028020 10771 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
08:16:19.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.030890 10771 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/data/instance:
08:16:19.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "f2a3f45a9e3b4f2dbda7e2613df89453"
08:16:19.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:16:19 on dist-test-slave-gp6n"
08:16:19.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.031904 10771 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/wal/instance:
08:16:19.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "f2a3f45a9e3b4f2dbda7e2613df89453"
08:16:19.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:16:19 on dist-test-slave-gp6n"
08:16:19.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.038156 10771 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.002s	sys 0.004s
08:16:19.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.041990 10788 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:16:19.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.043421 10771 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
08:16:19.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.043622 10771 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/data,/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/wal
08:16:19.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "f2a3f45a9e3b4f2dbda7e2613df89453"
08:16:19.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:16:19 on dist-test-slave-gp6n"
08:16:19.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.043900 10771 fs_report.cc:389] FS layout report
08:16:19.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:16:19.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/wal
08:16:19.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/wal
08:16:19.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/data/data
08:16:19.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:16:19.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:16:19.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:16:19.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:16:19.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:16:19.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:16:19.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:16:19.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:16:19.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:16:19.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:16:19.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:16:19.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.057332 10771 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:16:19.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.058185 10771 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:16:19.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.058478 10771 kserver.cc:163] Server-wide thread pool size limit: 3276
08:16:19.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.059434 10771 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
08:16:19.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.061125 10771 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
08:16:19.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.061216 10771 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
08:16:19.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.061290 10771 ts_tablet_manager.cc:616] Registered 0 tablets
08:16:19.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.061343 10771 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
08:16:19.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.097291 10771 rpc_server.cc:307] RPC server started. Bound to: 127.10.40.65:39249
08:16:19.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.097327 10937 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.40.65:39249 every 8 connection(s)
08:16:19.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.099231 10771 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/data/info.pb
08:16:19.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.107650 10401 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0lHOBl/build/asan/bin/kudu as pid 10771
08:16:19.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.107952 10401 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/wal/instance
08:16:19.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.126300 10401 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0lHOBl/build/asan/bin/kudu
08:16:19.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task0lHOBl/build/asan/bin/kudu
08:16:19.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/wal
08:16:19.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/data
08:16:19.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:16:19.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:16:19.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:16:19.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:16:19.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:16:19.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:16:19.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/logs
08:16:19.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/data/info.pb
08:16:19.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:16:19.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:16:19.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:16:19.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:16:19.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:16:19.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:16:19.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:16:19.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:16:19.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:16:19.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:16:19.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.40.66:0
08:16:19.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.40.66
08:16:19.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.40.66
08:16:19.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
08:16:19.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.40.126:45793,127.10.40.125:46051,127.10.40.124:41773
08:16:19.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.40.84:40955
08:16:19.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:16:19.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:16:19.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:16:19.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.139431 10938 heartbeater.cc:344] Connected to a master server at 127.10.40.124:41773
08:16:19.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.139468 10939 heartbeater.cc:344] Connected to a master server at 127.10.40.126:45793
08:16:19.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.139864 10938 heartbeater.cc:461] Registering TS with master...
08:16:19.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.139868 10939 heartbeater.cc:461] Registering TS with master...
08:16:19.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.141041 10938 heartbeater.cc:507] Master 127.10.40.124:41773 requested a full tablet report, sending...
08:16:19.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.141850 10940 heartbeater.cc:344] Connected to a master server at 127.10.40.125:46051
08:16:19.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.142014 10940 heartbeater.cc:461] Registering TS with master...
08:16:19.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.142581 10940 heartbeater.cc:507] Master 127.10.40.125:46051 requested a full tablet report, sending...
08:16:19.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.143162 10733 catalog_manager.cc:1295] Loaded cluster ID: db600596164f4994a1198e869813dd76
08:16:19.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.143260 10733 catalog_manager.cc:1588] T 00000000000000000000000000000000 P a0b587403b6a4d3090cfced49037efc0: loading cluster ID for follower catalog manager: success
08:16:19.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.145401 10550 ts_manager.cc:194] Registered new tserver with Master: f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65:39249)
08:16:19.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.146790 10939 heartbeater.cc:507] Master 127.10.40.126:45793 requested a full tablet report, sending...
08:16:19.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.147604 10733 catalog_manager.cc:1610] T 00000000000000000000000000000000 P a0b587403b6a4d3090cfced49037efc0: acquiring CA information for follower catalog manager: success
08:16:19.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.148922 10648 ts_manager.cc:194] Registered new tserver with Master: f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65:39249)
08:16:19.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.149045 10451 ts_manager.cc:194] Registered new tserver with Master: f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65:39249)
08:16:19.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.150861 10733 catalog_manager.cc:1638] T 00000000000000000000000000000000 P a0b587403b6a4d3090cfced49037efc0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
08:16:19.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.151517 10648 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.10.40.65:53809
08:16:19.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.279937 10755 catalog_manager.cc:1295] Loaded cluster ID: db600596164f4994a1198e869813dd76
08:16:19.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.280085 10755 catalog_manager.cc:1588] T 00000000000000000000000000000000 P b7c1f40057e3404093b0ed154254604f: loading cluster ID for follower catalog manager: success
08:16:19.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.283257 10755 catalog_manager.cc:1610] T 00000000000000000000000000000000 P b7c1f40057e3404093b0ed154254604f: acquiring CA information for follower catalog manager: success
08:16:19.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.285344 10755 catalog_manager.cc:1638] T 00000000000000000000000000000000 P b7c1f40057e3404093b0ed154254604f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
08:16:19.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:19.484745 10948 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:16:19.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:19.485064 10948 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:16:19.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:19.485137 10948 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:16:19.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:19.493634 10948 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:16:19.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:19.493806 10948 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.40.66
08:16:19.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.509569 10948 tablet_server_runner.cc:78] Tablet server non-default flags:
08:16:19.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/logs
08:16:19.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:16:19.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:16:19.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:16:19.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.40.84:40955
08:16:19.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:16:19.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:16:19.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/data
08:16:19.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/wal
08:16:19.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:16:19.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:16:19.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.40.66:0
08:16:19.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:16:19.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:16:19.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:16:19.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/data/info.pb
08:16:19.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.40.66
08:16:19.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
08:16:19.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.40.126:45793,127.10.40.125:46051,127.10.40.124:41773
08:16:19.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:16:19.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:16:19.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:16:19.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:16:19.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:16:19.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.40.66
08:16:19.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:16:19.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:16:19.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:16:19.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 70c51312f1186a8ae9f46647160b7730123f7601
08:16:19.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:16:19.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 12 Aug 2026 07:43:21 UTC on bdcb31816ec0
08:16:19.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 13302
08:16:19.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
08:16:19.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.511809 10948 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:16:19.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.514099 10948 file_cache.cc:504] Constructed file cache file cache with capacity 419430
08:16:19.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:19.524772 10954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:16:19.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:19.524782 10953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:16:19.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:19.525432 10957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:16:19.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.526072 10948 server_base.cc:1061] running on GCE node
08:16:19.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.526711 10948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:16:19.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.527673 10948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:16:19.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.528935 10948 hybrid_clock.cc:648] HybridClock initialized: now 1786522579528880 us; error 38 us; skew 500 ppm
08:16:19.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.529318 10948 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:16:19.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.531695 10948 webserver.cc:533] Webserver started at http://127.10.40.66:33605/ using document root <none> and password file <none>
08:16:19.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.532497 10948 fs_manager.cc:362] Metadata directory not provided
08:16:19.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.532644 10948 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:16:19.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.533072 10948 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
08:16:19.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.535776 10948 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/data/instance:
08:16:19.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "f0e29bd27cb448589ac37e569ad1a164"
08:16:19.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:16:19 on dist-test-slave-gp6n"
08:16:19.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.536726 10948 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/wal/instance:
08:16:19.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "f0e29bd27cb448589ac37e569ad1a164"
08:16:19.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:16:19 on dist-test-slave-gp6n"
08:16:19.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.542312 10948 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.000s
08:16:19.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.545766 10965 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:16:19.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.547214 10948 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
08:16:19.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.547619 10948 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/data,/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/wal
08:16:19.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "f0e29bd27cb448589ac37e569ad1a164"
08:16:19.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:16:19 on dist-test-slave-gp6n"
08:16:19.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.547904 10948 fs_report.cc:389] FS layout report
08:16:19.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:16:19.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/wal
08:16:19.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/wal
08:16:19.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/data/data
08:16:19.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:16:19.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:16:19.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:16:19.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:16:19.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:16:19.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:16:19.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:16:19.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:16:19.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:16:19.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:16:19.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:16:19.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.574877 10948 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:16:19.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.575836 10948 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:16:19.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.576087 10948 kserver.cc:163] Server-wide thread pool size limit: 3276
08:16:19.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.577055 10948 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
08:16:19.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.578974 10948 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
08:16:19.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.579066 10948 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
08:16:19.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.579140 10948 ts_tablet_manager.cc:616] Registered 0 tablets
08:16:19.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.579192 10948 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
08:16:19.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.614737 10948 rpc_server.cc:307] RPC server started. Bound to: 127.10.40.66:46247
08:16:19.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.614795 11104 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.40.66:46247 every 8 connection(s)
08:16:19.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.616458 10948 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/data/info.pb
08:16:19.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.618733 10401 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0lHOBl/build/asan/bin/kudu as pid 10948
08:16:19.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.618947 10401 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/wal/instance
08:16:19.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.624464 10401 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0lHOBl/build/asan/bin/kudu
08:16:19.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task0lHOBl/build/asan/bin/kudu
08:16:19.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/wal
08:16:19.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/data
08:16:19.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:16:19.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:16:19.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:16:19.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:16:19.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:16:19.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:16:19.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/logs
08:16:19.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/data/info.pb
08:16:19.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:16:19.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:16:19.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:16:19.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:16:19.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:16:19.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:16:19.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:16:19.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:16:19.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:16:19.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:16:19.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.40.67:0
08:16:19.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.40.67
08:16:19.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.40.67
08:16:19.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
08:16:19.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.40.126:45793,127.10.40.125:46051,127.10.40.124:41773
08:16:19.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.40.84:40955
08:16:19.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:16:19.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:16:19.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:16:19.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.644127 11107 heartbeater.cc:344] Connected to a master server at 127.10.40.126:45793
08:16:19.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.644572 11107 heartbeater.cc:461] Registering TS with master...
08:16:19.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.645558 11105 heartbeater.cc:344] Connected to a master server at 127.10.40.124:41773
08:16:19.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.645706 11105 heartbeater.cc:461] Registering TS with master...
08:16:19.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.645802 11107 heartbeater.cc:507] Master 127.10.40.126:45793 requested a full tablet report, sending...
08:16:19.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.646284 11105 heartbeater.cc:507] Master 127.10.40.124:41773 requested a full tablet report, sending...
08:16:19.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.647732 10648 ts_manager.cc:194] Registered new tserver with Master: f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247)
08:16:19.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.647797 10451 ts_manager.cc:194] Registered new tserver with Master: f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247)
08:16:19.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.648974 10648 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.10.40.66:54969
08:16:19.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.652293 11108 heartbeater.cc:344] Connected to a master server at 127.10.40.125:46051
08:16:19.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.652467 11108 heartbeater.cc:461] Registering TS with master...
08:16:19.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.652855 11108 heartbeater.cc:507] Master 127.10.40.125:46051 requested a full tablet report, sending...
08:16:19.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:19.654158 10550 ts_manager.cc:194] Registered new tserver with Master: f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247)
08:16:19.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:19.996315 11113 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:16:19.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:19.996699 11113 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:16:19.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:19.996806 11113 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:16:20.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:20.005512 11113 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:16:20.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:20.005741 11113 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.40.67
08:16:20.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:20.016300 11113 tablet_server_runner.cc:78] Tablet server non-default flags:
08:16:20.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/logs
08:16:20.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:16:20.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:16:20.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:16:20.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.40.84:40955
08:16:20.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:16:20.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:16:20.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/data
08:16:20.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/wal
08:16:20.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:16:20.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:16:20.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.40.67:0
08:16:20.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:16:20.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:16:20.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:16:20.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/data/info.pb
08:16:20.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.40.67
08:16:20.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
08:16:20.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.40.126:45793,127.10.40.125:46051,127.10.40.124:41773
08:16:20.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:16:20.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:16:20.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:16:20.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:16:20.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:16:20.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.40.67
08:16:20.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:16:20.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:16:20.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:16:20.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 70c51312f1186a8ae9f46647160b7730123f7601
08:16:20.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:16:20.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 12 Aug 2026 07:43:21 UTC on bdcb31816ec0
08:16:20.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 13302
08:16:20.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
08:16:20.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:20.018718 11113 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:16:20.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:20.020851 11113 file_cache.cc:504] Constructed file cache file cache with capacity 419430
08:16:20.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:20.031032 11124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:16:20.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:20.031028 11125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:16:20.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:20.031929 11127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:16:20.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:20.032548 11113 server_base.cc:1061] running on GCE node
08:16:20.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:20.033172 11113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:16:20.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:20.034173 11113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:16:20.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:20.035386 11113 hybrid_clock.cc:648] HybridClock initialized: now 1786522580035341 us; error 61 us; skew 500 ppm
08:16:20.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:20.035671 11113 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:16:20.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:20.037861 11113 webserver.cc:533] Webserver started at http://127.10.40.67:45387/ using document root <none> and password file <none>
08:16:20.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:20.038722 11113 fs_manager.cc:362] Metadata directory not provided
08:16:20.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:20.038826 11113 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:16:20.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:20.039155 11113 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
08:16:20.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:20.041675 11113 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/data/instance:
08:16:20.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "2ea1996689fb4cf18b17a35790fba392"
08:16:20.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:16:20 on dist-test-slave-gp6n"
08:16:20.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:20.042466 11113 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/wal/instance:
08:16:20.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "2ea1996689fb4cf18b17a35790fba392"
08:16:20.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:16:20 on dist-test-slave-gp6n"
08:16:20.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:20.048041 11113 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
08:16:20.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:20.051843 11134 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:16:20.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:20.053272 11113 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
08:16:20.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:20.053462 11113 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/data,/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/wal
08:16:20.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "2ea1996689fb4cf18b17a35790fba392"
08:16:20.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:16:20 on dist-test-slave-gp6n"
08:16:20.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:20.053727 11113 fs_report.cc:389] FS layout report
08:16:20.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:16:20.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/wal
08:16:20.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/wal
08:16:20.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/data/data
08:16:20.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:16:20.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:16:20.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:16:20.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:16:20.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:16:20.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:16:20.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:16:20.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:16:20.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:16:20.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:16:20.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:16:20.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:20.093922 11113 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:16:20.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:20.094841 11113 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:16:20.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:20.095093 11113 kserver.cc:163] Server-wide thread pool size limit: 3276
08:16:20.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:20.096057 11113 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
08:16:20.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:20.098269 11113 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
08:16:20.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:20.098418 11113 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
08:16:20.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:20.098528 11113 ts_tablet_manager.cc:616] Registered 0 tablets
08:16:20.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:20.098603 11113 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
08:16:20.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:20.151115 11113 rpc_server.cc:307] RPC server started. Bound to: 127.10.40.67:44983
08:16:20.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:20.151211 11286 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.40.67:44983 every 8 connection(s)
08:16:20.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:20.153097 11113 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/data/info.pb
08:16:20.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:20.154253 10938 heartbeater.cc:499] Master 127.10.40.124:41773 was elected leader, sending a full tablet report...
08:16:20.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:20.157068 10401 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0lHOBl/build/asan/bin/kudu as pid 11113
08:16:20.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:20.157253 10401 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/wal/instance
08:16:20.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:20.188233 11287 heartbeater.cc:344] Connected to a master server at 127.10.40.124:41773
08:16:20.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:20.188673 11287 heartbeater.cc:461] Registering TS with master...
08:16:20.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:20.189838 11287 heartbeater.cc:507] Master 127.10.40.124:41773 requested a full tablet report, sending...
08:16:20.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:20.190552 11288 heartbeater.cc:344] Connected to a master server at 127.10.40.126:45793
08:16:20.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:20.190743 11288 heartbeater.cc:461] Registering TS with master...
08:16:20.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:20.191435 11288 heartbeater.cc:507] Master 127.10.40.126:45793 requested a full tablet report, sending...
08:16:20.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:20.191752 10648 ts_manager.cc:194] Registered new tserver with Master: 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67:44983)
08:16:20.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:20.192562 10451 ts_manager.cc:194] Registered new tserver with Master: 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67:44983)
08:16:20.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:20.192869 10648 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.10.40.67:48219
08:16:20.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:20.194172 11289 heartbeater.cc:344] Connected to a master server at 127.10.40.125:46051
08:16:20.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:20.194309 11289 heartbeater.cc:461] Registering TS with master...
08:16:20.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:20.194840 11289 heartbeater.cc:507] Master 127.10.40.125:46051 requested a full tablet report, sending...
08:16:20.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:20.195838 10550 ts_manager.cc:194] Registered new tserver with Master: 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67:44983)
08:16:20.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:20.202683 10401 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
08:16:20.204 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
08:16:20.205 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_masters {
}

08:16:20.215 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_masters {
  masters {
    id {
      type: MASTER
      index: 0
    }
    bound_rpc_address {
      host: "127.10.40.126"
      port: 45793
    }
    bound_http_address {
      host: "127.10.40.126"
      port: 34815
    }
  }
  masters {
    id {
      type: MASTER
      index: 1
    }
    bound_rpc_address {
      host: "127.10.40.125"
      port: 46051
    }
    bound_http_address {
      host: "127.10.40.125"
      port: 43479
    }
  }
  masters {
    id {
      type: MASTER
      index: 2
    }
    bound_rpc_address {
      host: "127.10.40.124"
      port: 41773
    }
    bound_http_address {
      host: "127.10.40.124"
      port: 41359
    }
  }
}

08:16:20.219 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_tservers {
}

08:16:20.222 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_tservers {
  tservers {
    id {
      type: TSERVER
      index: 0
    }
    bound_rpc_address {
      host: "127.10.40.65"
      port: 39249
    }
    bound_http_address {
      host: "127.10.40.65"
      port: 41939
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 1
    }
    bound_rpc_address {
      host: "127.10.40.66"
      port: 46247
    }
    bound_http_address {
      host: "127.10.40.66"
      port: 33605
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 2
    }
    bound_rpc_address {
      host: "127.10.40.67"
      port: 44983
    }
    bound_http_address {
      host: "127.10.40.67"
      port: 45387
    }
  }
}

08:16:20.223 [INFO - main] (KuduTestHarness.java:157) Creating a new Kudu client...
08:16:20.453 [DEBUG - main] (CompositeStrategy.java:59) Unable to load modern Subject methods, relying only on legacy methods
java.lang.NoSuchMethodException: javax.security.auth.Subject.current()
	at java.base/java.lang.Class.getDeclaredMethod(Class.java:2677)
	at org.apache.kudu.client.internals.ModernStrategy.<init>(ModernStrategy.java:51)
	at org.apache.kudu.client.internals.CompositeStrategy.<init>(CompositeStrategy.java:52)
	at org.apache.kudu.client.internals.CompositeStrategy.<clinit>(CompositeStrategy.java:41)
	at org.apache.kudu.client.internals.SecurityManagerCompatibility.get(SecurityManagerCompatibility.java:43)
	at org.apache.kudu.client.SecurityContext.setupSubject(SecurityContext.java:155)
	at org.apache.kudu.client.SecurityContext.<init>(SecurityContext.java:138)
	at org.apache.kudu.client.AsyncKuduClient.<init>(AsyncKuduClient.java:431)
	at org.apache.kudu.client.AsyncKuduClient$AsyncKuduClientBuilder.build(AsyncKuduClient.java:3109)
	at org.apache.kudu.test.KuduTestHarness.before(KuduTestHarness.java:161)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:50)
	at org.apache.kudu.test.junit.RetryRule$RetryStatement.doOneAttemptAndReport(RetryRule.java:184)
	at org.apache.kudu.test.junit.RetryRule$RetryStatement.evaluate(RetryRule.java:244)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:27)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
	at org.junit.runner.JUnitCore.runMain(JUnitCore.java:77)
	at org.junit.runner.JUnitCore.main(JUnitCore.java:36)
08:16:20.524 [DEBUG - main] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 
08:16:20.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:20.652005 11105 heartbeater.cc:499] Master 127.10.40.124:41773 was elected leader, sending a full tablet report...
08:16:20.698 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.10.40.124:41773(127.10.40.124:41773)] connecting to peer
08:16:20.752 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.10.40.126:45793(127.10.40.126:45793)] connecting to peer
08:16:20.754 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.10.40.125:46051(127.10.40.125:46051)] connecting to peer
08:16:20.773 [DEBUG - kudu-nio-2] (Connection.java:757) [peer master-127.10.40.125:46051(127.10.40.125:46051)] Successfully connected to peer
08:16:20.776 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.10.40.126:45793(127.10.40.126:45793)] Successfully connected to peer
08:16:20.781 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.10.40.124:41773(127.10.40.124:41773)] Successfully connected to peer
08:16:20.866 [DEBUG - kudu-nio-0] (Negotiator.java:528) SASL mechanism PLAIN chosen for peer 127.10.40.124
08:16:20.868 [DEBUG - kudu-nio-1] (Negotiator.java:528) SASL mechanism PLAIN chosen for peer 127.10.40.126
08:16:20.874 [DEBUG - kudu-nio-2] (Negotiator.java:528) SASL mechanism PLAIN chosen for peer 127.10.40.125
08:16:21.103 [DEBUG - kudu-nio-2] (Negotiator.java:918) Authenticated connection [id: 0xc662a025, L:/127.0.0.1:42976 - R:/127.10.40.125:46051] using SASL/PLAIN
08:16:21.103 [DEBUG - kudu-nio-1] (Negotiator.java:918) Authenticated connection [id: 0xef710055, L:/127.0.0.1:54452 - R:/127.10.40.126:45793] using SASL/PLAIN
08:16:21.103 [DEBUG - kudu-nio-0] (Negotiator.java:918) Authenticated connection [id: 0xd02305bc, L:/127.0.0.1:43142 - R:/127.10.40.124:41773] using SASL/PLAIN
08:16:21.138 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet Kudu Master for table 'Kudu Master' with partition [<start>, <end>)
08:16:21.139 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=CreateTable, tablet=null, attempt=1, TimeoutTracker(timeout=50000, elapsed=477), Traces: [0ms] refreshing cache from master, [22ms] Sub RPC ConnectToMaster: sending RPC to server master-127.10.40.124:41773, [86ms] Sub RPC ConnectToMaster: sending RPC to server master-127.10.40.126:45793, [87ms] Sub RPC ConnectToMaster: sending RPC to server master-127.10.40.125:46051, [450ms] Sub RPC ConnectToMaster: received response from server master-127.10.40.125:46051: OK, [450ms] Sub RPC ConnectToMaster: received response from server master-127.10.40.126:45793: OK, [451ms] Sub RPC ConnectToMaster: received response from server master-127.10.40.124:41773: OK, deferred=null) after lookup
08:16:21.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.189957 10648 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43142:
08:16:21.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) name: "org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644"
08:16:21.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) schema {
08:16:21.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
08:16:21.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "key"
08:16:21.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
08:16:21.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: true
08:16:21.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
08:16:21.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
08:16:21.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
08:16:21.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
08:16:21.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
08:16:21.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
08:16:21.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column1_i"
08:16:21.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
08:16:21.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
08:16:21.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
08:16:21.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
08:16:21.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
08:16:21.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
08:16:21.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
08:16:21.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
08:16:21.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column2_i"
08:16:21.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
08:16:21.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
08:16:21.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
08:16:21.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
08:16:21.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
08:16:21.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
08:16:21.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
08:16:21.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
08:16:21.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column3_s"
08:16:21.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: STRING
08:16:21.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
08:16:21.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: true
08:16:21.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     encoding: DICT_ENCODING
08:16:21.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     compression: LZ4
08:16:21.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 4096
08:16:21.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
08:16:21.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
08:16:21.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
08:16:21.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
08:16:21.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column4_b"
08:16:21.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: BOOL
08:16:21.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
08:16:21.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
08:16:21.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
08:16:21.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
08:16:21.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
08:16:21.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
08:16:21.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
08:16:21.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) partition_schema {
08:16:21.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   range_schema {
08:16:21.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     columns {
08:16:21.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)       name: "key"
08:16:21.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     }
08:16:21.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
08:16:21.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
08:16:21.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:21.191850 10648 catalog_manager.cc:7099] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
08:16:21.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.195768 11287 heartbeater.cc:499] Master 127.10.40.124:41773 was elected leader, sending a full tablet report...
08:16:21.244 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:894) Opened table 562856cf613c444685978a2814537f6d
08:16:21.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.269995 11023 tablet_service.cc:1511] Processing CreateTablet for tablet b328961edc164a1289f259240593e73e (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644 [id=562856cf613c444685978a2814537f6d]), partition=RANGE (key) PARTITION UNBOUNDED
08:16:21.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.275533 11023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b328961edc164a1289f259240593e73e. 1 dirs total, 0 dirs full, 0 dirs failed
08:16:21.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.275202 10851 tablet_service.cc:1511] Processing CreateTablet for tablet b328961edc164a1289f259240593e73e (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644 [id=562856cf613c444685978a2814537f6d]), partition=RANGE (key) PARTITION UNBOUNDED
08:16:21.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.275768 11201 tablet_service.cc:1511] Processing CreateTablet for tablet b328961edc164a1289f259240593e73e (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644 [id=562856cf613c444685978a2814537f6d]), partition=RANGE (key) PARTITION UNBOUNDED
08:16:21.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.276779 10851 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b328961edc164a1289f259240593e73e. 1 dirs total, 0 dirs full, 0 dirs failed
08:16:21.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.277391 11201 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b328961edc164a1289f259240593e73e. 1 dirs total, 0 dirs full, 0 dirs failed
08:16:21.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.291364 11336 tablet_bootstrap.cc:492] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164: Bootstrap starting.
08:16:21.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.295647 11336 tablet_bootstrap.cc:654] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164: Neither blocks nor log segments found. Creating new log.
08:16:21.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.297272 11336 log.cc:826] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164: Log is configured to *not* fsync() on all Append() calls
08:16:21.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.302734 11336 tablet_bootstrap.cc:492] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164: No bootstrap required, opened a new log
08:16:21.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.309717 11336 ts_tablet_manager.cc:1403] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164: Time spent bootstrapping tablet: real 0.019s	user 0.010s	sys 0.000s
08:16:21.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.316181 11337 tablet_bootstrap.cc:492] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453: Bootstrap starting.
08:16:21.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.317298 11338 tablet_bootstrap.cc:492] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392: Bootstrap starting.
08:16:21.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.321607 11338 tablet_bootstrap.cc:654] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392: Neither blocks nor log segments found. Creating new log.
08:16:21.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.321954 11337 tablet_bootstrap.cc:654] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453: Neither blocks nor log segments found. Creating new log.
08:16:21.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.321910 11336 raft_consensus.cc:359] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:21.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.322638 11336 raft_consensus.cc:385] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
08:16:21.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.323335 11338 log.cc:826] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392: Log is configured to *not* fsync() on all Append() calls
08:16:21.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.323566 11337 log.cc:826] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453: Log is configured to *not* fsync() on all Append() calls
08:16:21.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.330150 11336 raft_consensus.cc:740] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e29bd27cb448589ac37e569ad1a164, State: Initialized, Role: FOLLOWER
08:16:21.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.331557 11336 consensus_queue.cc:260] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:21.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.333189 11337 tablet_bootstrap.cc:492] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453: No bootstrap required, opened a new log
08:16:21.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.333705 11337 ts_tablet_manager.cc:1403] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453: Time spent bootstrapping tablet: real 0.018s	user 0.015s	sys 0.000s
08:16:21.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.334271 11338 tablet_bootstrap.cc:492] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392: No bootstrap required, opened a new log
08:16:21.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.334910 11338 ts_tablet_manager.cc:1403] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392: Time spent bootstrapping tablet: real 0.018s	user 0.015s	sys 0.000s
08:16:21.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.345134 11338 raft_consensus.cc:359] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:21.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.346539 11338 raft_consensus.cc:385] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
08:16:21.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.346866 11338 raft_consensus.cc:740] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ea1996689fb4cf18b17a35790fba392, State: Initialized, Role: FOLLOWER
08:16:21.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.348596 11336 ts_tablet_manager.cc:1434] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164: Time spent starting tablet: real 0.038s	user 0.016s	sys 0.008s
08:16:21.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.354768 11338 consensus_queue.cc:260] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:21.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.357625 11337 raft_consensus.cc:359] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:21.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.358321 11337 raft_consensus.cc:385] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
08:16:21.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.358649 11337 raft_consensus.cc:740] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2a3f45a9e3b4f2dbda7e2613df89453, State: Initialized, Role: FOLLOWER
08:16:21.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.359700 11337 consensus_queue.cc:260] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:21.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.362183 11338 ts_tablet_manager.cc:1434] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392: Time spent starting tablet: real 0.027s	user 0.013s	sys 0.004s
08:16:21.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.372819 11337 ts_tablet_manager.cc:1434] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453: Time spent starting tablet: real 0.030s	user 0.010s	sys 0.008s
08:16:21.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:21.381860 11110 tablet.cc:2406] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164: Can't schedule compaction. Clean time has not been advanced past its initial value.
08:16:21.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:21.430819 11297 tablet.cc:2406] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392: Can't schedule compaction. Clean time has not been advanced past its initial value.
08:16:21.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.595566 11344 raft_consensus.cc:493] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
08:16:21.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.595831 11344 raft_consensus.cc:515] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:21.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.597786 11344 leader_election.cc:290] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247), f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65:39249)
08:16:21.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:21.605274 10941 tablet.cc:2406] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453: Can't schedule compaction. Clean time has not been advanced past its initial value.
08:16:21.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.606887 11049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b328961edc164a1289f259240593e73e" candidate_uuid: "2ea1996689fb4cf18b17a35790fba392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e29bd27cb448589ac37e569ad1a164" is_pre_election: true
08:16:21.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.607522 11049 raft_consensus.cc:2468] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ea1996689fb4cf18b17a35790fba392 in term 0.
08:16:21.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.607831 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b328961edc164a1289f259240593e73e" candidate_uuid: "2ea1996689fb4cf18b17a35790fba392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" is_pre_election: true
08:16:21.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.608355 10874 raft_consensus.cc:2468] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ea1996689fb4cf18b17a35790fba392 in term 0.
08:16:21.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.608297 11135 leader_election.cc:304] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ea1996689fb4cf18b17a35790fba392, f0e29bd27cb448589ac37e569ad1a164; no voters: 
08:16:21.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.608734 11344 raft_consensus.cc:2804] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 0 FOLLOWER]: Leader pre-election won for term 1
08:16:21.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.608891 11344 raft_consensus.cc:493] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
08:16:21.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.608986 11344 raft_consensus.cc:3060] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 0 FOLLOWER]: Advancing to term 1
08:16:21.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.611636 11344 raft_consensus.cc:515] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:21.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.612490 11344 leader_election.cc:290] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [CANDIDATE]: Term 1 election: Requested vote from peers f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247), f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65:39249)
08:16:21.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.613241 11049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b328961edc164a1289f259240593e73e" candidate_uuid: "2ea1996689fb4cf18b17a35790fba392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0e29bd27cb448589ac37e569ad1a164"
08:16:21.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.613308 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b328961edc164a1289f259240593e73e" candidate_uuid: "2ea1996689fb4cf18b17a35790fba392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453"
08:16:21.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.613468 11049 raft_consensus.cc:3060] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 0 FOLLOWER]: Advancing to term 1
08:16:21.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.613492 10874 raft_consensus.cc:3060] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 0 FOLLOWER]: Advancing to term 1
08:16:21.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.616981 11049 raft_consensus.cc:2468] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ea1996689fb4cf18b17a35790fba392 in term 1.
08:16:21.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.617121 10874 raft_consensus.cc:2468] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ea1996689fb4cf18b17a35790fba392 in term 1.
08:16:21.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.617645 11135 leader_election.cc:304] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ea1996689fb4cf18b17a35790fba392, f0e29bd27cb448589ac37e569ad1a164; no voters: 
08:16:21.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.617988 11344 raft_consensus.cc:2804] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 1 FOLLOWER]: Leader election won for term 1
08:16:21.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.618614 11344 raft_consensus.cc:697] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 1 LEADER]: Becoming Leader. State: Replica: 2ea1996689fb4cf18b17a35790fba392, State: Running, Role: LEADER
08:16:21.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.619676 11344 consensus_queue.cc:237] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:21.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:21.625296 10648 catalog_manager.cc:5719] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67). New cstate: current_term: 1 leader_uuid: "2ea1996689fb4cf18b17a35790fba392" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } health_report { overall_health: UNKNOWN } } }
08:16:21.890 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:894) Opened table 562856cf613c444685978a2814537f6d
08:16:21.908 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:21.914 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299994, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)[L],f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247),f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)]}]
08:16:21.915 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=null, attempt=1, sequence_id=-1, TimeoutTracker(timeout=30000, elapsed=17), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [8ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=null) after lookup
08:16:21.925 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] connecting to peer
08:16:21.928 [DEBUG - kudu-nio-3] (Connection.java:757) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] Successfully connected to peer
08:16:21.995 [DEBUG - kudu-nio-3] (Negotiator.java:918) Authenticated connection [id: 0x3c8b9e02, L:/127.0.0.1:45556 - R:/127.10.40.67:44983] using TOKEN/null
08:16:22.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:22.008587 10874 raft_consensus.cc:1275] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 1 FOLLOWER]: Refusing update from remote peer 2ea1996689fb4cf18b17a35790fba392: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
08:16:22.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:22.008584 11049 raft_consensus.cc:1275] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 1 FOLLOWER]: Refusing update from remote peer 2ea1996689fb4cf18b17a35790fba392: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
08:16:22.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:22.009805 11344 consensus_queue.cc:1048] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
08:16:22.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:22.010134 11349 consensus_queue.cc:1048] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
08:16:22.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:22.024750 11355 mvcc.cc:204] Tried to move back new op lower bound from 7317596495894052864 to 7317596494319108096. Current Snapshot: MvccSnapshot[applied={T|T < 7317596495894052864}]
08:16:22.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:22.050361 11364 mvcc.cc:204] Tried to move back new op lower bound from 7317596495894052864 to 7317596494319108096. Current Snapshot: MvccSnapshot[applied={T|T < 7317596495894052864}]
08:16:22.150 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 3 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644, tablet=b328961edc164a1289f259240593e73e, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:16:22.152 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.10.40.67:44983
08:16:22.154 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

08:16:22.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:22.155085 10401 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-task0lHOBl/build/asan/bin/kudu pid 11113
08:16:22.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:22.185875 11279 generic_service.cc:196] Checking for leaks (request via RPC)
08:16:22.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:22.290767 10401 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0lHOBl/build/asan/bin/kudu with pid 11113
08:16:22.317 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:16:22.319 [DEBUG - kudu-nio-3] (Connection.java:239) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] handling channelInactive
08:16:22.320 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] cleaning up while in state READY due to: connection closed
08:16:22.320 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983) for tablet b328961edc164a1289f259240593e73e: connection closed
08:16:22.327 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:22.327 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)[L],f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247),f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)]}]
08:16:22.328 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=2, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=10), Traces: [0ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [3ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [10ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@420087740(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@38b26d4d -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@677adce6 -> wakeup thread Time-limited test)) after lookup
08:16:22.328 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] connecting to peer
08:16:22.331 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:22.331 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] cleaning up while in state CONNECTING due to: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:22.331 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:22.346 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:22.347 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)[L],f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247),f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)]}]
08:16:22.347 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=4, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=29), Traces: [0ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [3ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [10ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [11ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [14ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [14ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [29ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@420087740(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@38b26d4d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1978709144 after retry RPC@1564496738), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@677adce6 -> wakeup thread Time-limited test -> (continuation of Deferred@1978709144 after retry RPC@1564496738))) after lookup
08:16:22.348 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] connecting to peer
08:16:22.350 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:22.351 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] cleaning up while in state CONNECTING due to: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:22.351 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:22.386 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:22.387 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)[L],f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247),f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)]}]
08:16:22.388 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=6, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=70), Traces: [0ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [3ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [10ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [11ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [14ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [14ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [29ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [31ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [34ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [34ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [66ms] refreshing cache from master, [66ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [69ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@420087740(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@38b26d4d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1978709144 after retry RPC@1564496738) -> (continuation of Deferred@1201912276 after retry RPC@611262924), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@677adce6 -> wakeup thread Time-limited test -> (continuation of Deferred@1978709144 after retry RPC@1564496738) -> (continuation of Deferred@1201912276 after retry RPC@611262924))) after lookup
08:16:22.388 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] connecting to peer
08:16:22.390 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:22.390 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] cleaning up while in state CONNECTING due to: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:22.390 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:22.467 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:22.467 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)[L],f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247),f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)]}]
08:16:22.468 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=8, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=150), Traces: [0ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [3ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [10ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [11ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [14ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [14ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [29ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [31ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [34ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [34ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [66ms] refreshing cache from master, [66ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [69ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [71ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [73ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [73ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [146ms] refreshing cache from master, [146ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [149ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@420087740(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@38b26d4d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1978709144 after retry RPC@1564496738) -> (continuation of Deferred@1201912276 after retry RPC@611262924) -> (continuation of Deferred@1920923386 after retry RPC@119940207), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@677adce6 -> wakeup thread Time-limited test -> (continuation of Deferred@1978709144 after retry RPC@1564496738) -> (continuation of Deferred@1201912276 after retry RPC@611262924) -> (continuation of Deferred@1920923386 after retry RPC@119940207))) after lookup
08:16:22.469 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] connecting to peer
08:16:22.471 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:22.471 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] cleaning up while in state CONNECTING due to: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:22.472 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:22.907 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:22.908 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)[L],f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247),f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)]}]
08:16:22.908 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=10, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=590), Traces: [0ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [3ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [10ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [11ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [14ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [14ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [29ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [31ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [34ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [34ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [66ms] refreshing cache from master, [66ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [69ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [71ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [73ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [73ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [146ms] refreshing cache from master, [146ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [149ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [152ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [154ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [155ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [586ms] refreshing cache from master, [586ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [590ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@420087740(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@38b26d4d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1978709144 after retry RPC@1564496738) -> (continuation of Deferred@1201912276 after retry RPC@611262924) -> (continuation of Deferred@1920923386 after retry RPC@119940207) -> (continuation of Deferred@1432129444 after retry RPC@744021766), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@677adce6 -> wakeup thread Time-limited test -> (continuation of Deferred@1978709144 after retry RPC@1564496738) -> (continuation of Deferred@1201912276 after retry RPC@611262924) -> (continuation of Deferred@1920923386 after retry RPC@119940207) -> (continuation of Deferred@1432129444 after retry RPC@744021766))) after lookup
08:16:22.909 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] connecting to peer
08:16:22.910 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:22.910 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] cleaning up while in state CONNECTING due to: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:22.911 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:23.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:23.581583 11382 raft_consensus.cc:493] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2ea1996689fb4cf18b17a35790fba392)
08:16:23.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:23.581784 11382 raft_consensus.cc:515] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:23.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:23.581844 11383 raft_consensus.cc:493] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2ea1996689fb4cf18b17a35790fba392)
08:16:23.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:23.582067 11383 raft_consensus.cc:515] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:23.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:23.583405 11382 leader_election.cc:290] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247), 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67:44983)
08:16:23.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:23.585072 11383 leader_election.cc:290] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67:44983), f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65:39249)
08:16:23.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:23.586305 10790 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.40.67:44983: connect: Connection refused (error 111)
08:16:23.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:23.585187 10966 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.40.67:44983: connect: Connection refused (error 111)
08:16:23.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:23.589031 10790 leader_election.cc:336] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67:44983): Network error: Client connection negotiation failed: client connection to 127.10.40.67:44983: connect: Connection refused (error 111)
08:16:23.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:23.590584 10966 leader_election.cc:336] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67:44983): Network error: Client connection negotiation failed: client connection to 127.10.40.67:44983: connect: Connection refused (error 111)
08:16:23.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:23.592496 11049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b328961edc164a1289f259240593e73e" candidate_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0e29bd27cb448589ac37e569ad1a164" is_pre_election: true
08:16:23.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:23.592705 11049 raft_consensus.cc:2468] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2a3f45a9e3b4f2dbda7e2613df89453 in term 1.
08:16:23.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:23.593209 10790 leader_election.cc:304] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f0e29bd27cb448589ac37e569ad1a164, f2a3f45a9e3b4f2dbda7e2613df89453; no voters: 2ea1996689fb4cf18b17a35790fba392
08:16:23.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:23.593560 11382 raft_consensus.cc:2804] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 1 FOLLOWER]: Leader pre-election won for term 2
08:16:23.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:23.593691 11382 raft_consensus.cc:493] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2ea1996689fb4cf18b17a35790fba392)
08:16:23.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:23.593788 11382 raft_consensus.cc:3060] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 1 FOLLOWER]: Advancing to term 2
08:16:23.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:23.596068 10874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b328961edc164a1289f259240593e73e" candidate_uuid: "f0e29bd27cb448589ac37e569ad1a164" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" is_pre_election: true
08:16:23.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:23.596725 11382 raft_consensus.cc:515] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:23.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:23.597116 10874 raft_consensus.cc:2393] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0e29bd27cb448589ac37e569ad1a164 in current term 2: Already voted for candidate f2a3f45a9e3b4f2dbda7e2613df89453 in this term.
08:16:23.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:23.597342 11382 leader_election.cc:290] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [CANDIDATE]: Term 2 election: Requested vote from peers f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247), 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67:44983)
08:16:23.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:23.597713 11049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b328961edc164a1289f259240593e73e" candidate_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0e29bd27cb448589ac37e569ad1a164"
08:16:23.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:23.597930 11049 raft_consensus.cc:3060] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 1 FOLLOWER]: Advancing to term 2
08:16:23.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:23.597894 10966 leader_election.cc:304] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0e29bd27cb448589ac37e569ad1a164; no voters: 2ea1996689fb4cf18b17a35790fba392, f2a3f45a9e3b4f2dbda7e2613df89453
08:16:23.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:23.599074 10790 leader_election.cc:336] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67:44983): Network error: Client connection negotiation failed: client connection to 127.10.40.67:44983: connect: Connection refused (error 111)
08:16:23.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:23.600274 11049 raft_consensus.cc:2468] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2a3f45a9e3b4f2dbda7e2613df89453 in term 2.
08:16:23.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:23.600459 11383 raft_consensus.cc:2749] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
08:16:23.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:23.600816 10790 leader_election.cc:304] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f0e29bd27cb448589ac37e569ad1a164, f2a3f45a9e3b4f2dbda7e2613df89453; no voters: 2ea1996689fb4cf18b17a35790fba392
08:16:23.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:23.601131 11382 raft_consensus.cc:2804] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 2 FOLLOWER]: Leader election won for term 2
08:16:23.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:23.601575 11382 raft_consensus.cc:697] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 2 LEADER]: Becoming Leader. State: Replica: f2a3f45a9e3b4f2dbda7e2613df89453, State: Running, Role: LEADER
08:16:23.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:23.602106 11382 consensus_queue.cc:237] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:23.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:23.605978 10648 catalog_manager.cc:5719] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 reported cstate change: term changed from 1 to 2, leader changed from 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67) to f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65). New cstate: current_term: 2 leader_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } health_report { overall_health: HEALTHY } } }
08:16:24.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:24.064952 10790 consensus_peers.cc:597] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 -> Peer 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67:44983): Couldn't send request to peer 2ea1996689fb4cf18b17a35790fba392. Status: Network error: Client connection negotiation failed: client connection to 127.10.40.67:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
08:16:24.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:24.155016 11049 raft_consensus.cc:1275] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 2 FOLLOWER]: Refusing update from remote peer f2a3f45a9e3b4f2dbda7e2613df89453: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
08:16:24.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:24.155903 11382 consensus_queue.cc:1048] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
08:16:24.527 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:24.528 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983),f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247),f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)[L]]}]
08:16:24.529 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=12, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=2210), Traces: [0ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [3ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [10ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [11ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [14ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [14ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [26ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [29ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [31ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [34ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [34ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [66ms] refreshing cache from master, [66ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [69ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [71ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [73ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [73ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [146ms] refreshing cache from master, [146ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [149ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [152ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [154ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [155ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [586ms] refreshing cache from master, [586ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [590ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [592ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [593ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [594ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [2206ms] refreshing cache from master, [2206ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [2210ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@420087740(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@38b26d4d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1978709144 after retry RPC@1564496738) -> (continuation of Deferred@1201912276 after retry RPC@611262924) -> (continuation of Deferred@1920923386 after retry RPC@119940207) -> (continuation of Deferred@1432129444 after retry RPC@744021766) -> (continuation of Deferred@437155904 after retry RPC@627176593), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@677adce6 -> wakeup thread Time-limited test -> (continuation of Deferred@1978709144 after retry RPC@1564496738) -> (continuation of Deferred@1201912276 after retry RPC@611262924) -> (continuation of Deferred@1920923386 after retry RPC@119940207) -> (continuation of Deferred@1432129444 after retry RPC@744021766) -> (continuation of Deferred@437155904 after retry RPC@627176593))) after lookup
08:16:24.530 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] connecting to peer
08:16:24.531 [DEBUG - kudu-nio-1] (Connection.java:757) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] Successfully connected to peer
08:16:24.551 [DEBUG - kudu-nio-1] (Negotiator.java:918) Authenticated connection [id: 0x072dabd1, L:/127.0.0.1:50314 - R:/127.10.40.65:39249] using TOKEN/null
08:16:24.578 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.10.40.67:44983
08:16:24.579 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

08:16:24.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:24.580641 10401 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0lHOBl/build/asan/bin/kudu
08:16:24.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task0lHOBl/build/asan/bin/kudu
08:16:24.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/wal
08:16:24.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/data
08:16:24.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:16:24.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:16:24.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:16:24.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:16:24.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:16:24.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:16:24.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/logs
08:16:24.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/data/info.pb
08:16:24.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:16:24.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:16:24.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:16:24.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:16:24.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:16:24.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:16:24.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:16:24.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:16:24.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:16:24.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:16:24.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.40.67:44983
08:16:24.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.40.67
08:16:24.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.40.126:45793,127.10.40.125:46051,127.10.40.124:41773
08:16:24.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45387
08:16:24.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.40.67
08:16:24.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.40.84:40955
08:16:24.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:16:24.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:16:24.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:16:24.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:24.932911 11420 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:16:24.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:24.933284 11420 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:16:24.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:24.933408 11420 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:16:24.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:24.941951 11420 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:16:24.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:24.942180 11420 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.40.67
08:16:24.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:24.953001 11420 tablet_server_runner.cc:78] Tablet server non-default flags:
08:16:24.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/logs
08:16:24.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:16:24.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:16:24.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:16:24.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.40.84:40955
08:16:24.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:16:24.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:16:24.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/data
08:16:24.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/wal
08:16:24.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:16:24.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:16:24.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.40.67:44983
08:16:24.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:16:24.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:16:24.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:16:24.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/data/info.pb
08:16:24.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.40.67
08:16:24.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45387
08:16:24.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.40.126:45793,127.10.40.125:46051,127.10.40.124:41773
08:16:24.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:16:24.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:16:24.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:16:24.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:16:24.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:16:24.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.40.67
08:16:24.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:16:24.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:16:24.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:16:24.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 70c51312f1186a8ae9f46647160b7730123f7601
08:16:24.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:16:24.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 12 Aug 2026 07:43:21 UTC on bdcb31816ec0
08:16:24.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 13302
08:16:24.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
08:16:24.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:24.955391 11420 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:16:24.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:24.957492 11420 file_cache.cc:504] Constructed file cache file cache with capacity 419430
08:16:24.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:24.967617 11428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:16:24.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:24.967700 11429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:16:24.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:24.968647 11433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:16:24.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:24.969750 11420 server_base.cc:1061] running on GCE node
08:16:24.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:24.970376 11420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:16:24.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:24.971433 11420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:16:24.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:24.972682 11420 hybrid_clock.cc:648] HybridClock initialized: now 1786522584972615 us; error 54 us; skew 500 ppm
08:16:24.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:24.973049 11420 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:16:24.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:24.975359 11420 webserver.cc:533] Webserver started at http://127.10.40.67:45387/ using document root <none> and password file <none>
08:16:24.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:24.976192 11420 fs_manager.cc:362] Metadata directory not provided
08:16:24.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:24.976342 11420 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:16:24.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:24.982344 11420 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.002s	sys 0.001s
08:16:24.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:24.985404 11441 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:16:24.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:24.986901 11420 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.002s
08:16:24.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:24.987083 11420 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/data,/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/wal
08:16:24.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "2ea1996689fb4cf18b17a35790fba392"
08:16:24.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:16:20 on dist-test-slave-gp6n"
08:16:24.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:24.987900 11420 fs_report.cc:389] FS layout report
08:16:24.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:16:24.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/wal
08:16:24.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/wal
08:16:24.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/data/data
08:16:24.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:16:24.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:16:24.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:16:24.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:16:24.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:16:24.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:16:24.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:16:24.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:16:24.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:16:24.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:16:24.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:16:25.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:25.008275 11420 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:16:25.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:25.009479 11420 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:16:25.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:25.009866 11420 kserver.cc:163] Server-wide thread pool size limit: 3276
08:16:25.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:25.011188 11420 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
08:16:25.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:25.013375 11450 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
08:16:25.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:25.019366 11420 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
08:16:25.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:25.019529 11420 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.007s	user 0.001s	sys 0.000s
08:16:25.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:25.019671 11420 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
08:16:25.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:25.023001 11420 ts_tablet_manager.cc:616] Registered 1 tablets
08:16:25.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:25.023150 11420 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.003s	sys 0.000s
08:16:25.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:25.023627 11450 tablet_bootstrap.cc:492] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392: Bootstrap starting.
08:16:25.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:25.048660 11450 log.cc:826] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392: Log is configured to *not* fsync() on all Append() calls
08:16:25.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:25.064679 11450 tablet_bootstrap.cc:492] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
08:16:25.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:25.065336 11450 tablet_bootstrap.cc:492] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392: Bootstrap complete.
08:16:25.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:25.066900 11450 ts_tablet_manager.cc:1403] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392: Time spent bootstrapping tablet: real 0.044s	user 0.024s	sys 0.019s
08:16:25.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:25.073803 11420 rpc_server.cc:307] RPC server started. Bound to: 127.10.40.67:44983
08:16:25.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:25.073900 11597 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.40.67:44983 every 8 connection(s)
08:16:25.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:25.074154 11450 raft_consensus.cc:359] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:25.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:25.075042 11450 raft_consensus.cc:740] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ea1996689fb4cf18b17a35790fba392, State: Initialized, Role: FOLLOWER
08:16:25.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:25.075313 11420 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/data/info.pb
08:16:25.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:25.076124 11450 consensus_queue.cc:260] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:25.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:25.078105 11450 ts_tablet_manager.cc:1434] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392: Time spent starting tablet: real 0.011s	user 0.005s	sys 0.005s
08:16:25.085 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:16:25.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:25.085156 10401 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0lHOBl/build/asan/bin/kudu as pid 11420
08:16:25.117 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 6 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644, tablet=b328961edc164a1289f259240593e73e, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:16:25.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:25.123283 11599 heartbeater.cc:344] Connected to a master server at 127.10.40.126:45793
08:16:25.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:25.123677 11599 heartbeater.cc:461] Registering TS with master...
08:16:25.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:25.124816 11598 heartbeater.cc:344] Connected to a master server at 127.10.40.124:41773
08:16:25.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:25.124987 11599 heartbeater.cc:507] Master 127.10.40.126:45793 requested a full tablet report, sending...
08:16:25.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:25.125016 11598 heartbeater.cc:461] Registering TS with master...
08:16:25.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:25.125818 11598 heartbeater.cc:507] Master 127.10.40.124:41773 requested a full tablet report, sending...
08:16:25.127 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.10.40.65:39249
08:16:25.127 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

08:16:25.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:25.128144 10401 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-task0lHOBl/build/asan/bin/kudu pid 10771
08:16:25.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:25.128727 10451 ts_manager.cc:194] Re-registered known tserver with Master: 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67:44983)
08:16:25.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:25.132624 10647 ts_manager.cc:194] Re-registered known tserver with Master: 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67:44983)
08:16:25.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:25.134166 10647 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.10.40.67:44145
08:16:25.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:25.137220 11598 heartbeater.cc:499] Master 127.10.40.124:41773 was elected leader, sending a full tablet report...
08:16:25.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:25.140084 11601 heartbeater.cc:344] Connected to a master server at 127.10.40.125:46051
08:16:25.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:25.140249 11601 heartbeater.cc:461] Registering TS with master...
08:16:25.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:25.140563 11601 heartbeater.cc:507] Master 127.10.40.125:46051 requested a full tablet report, sending...
08:16:25.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:25.142259 10550 ts_manager.cc:194] Re-registered known tserver with Master: 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67:44983)
08:16:25.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:25.145033 10932 generic_service.cc:196] Checking for leaks (request via RPC)
08:16:25.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:25.248405 10401 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0lHOBl/build/asan/bin/kudu with pid 10771
08:16:25.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:25.274685 10433 connection.cc:573] server connection from 127.10.40.65:45031 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
08:16:25.275 [DEBUG - kudu-nio-1] (Connection.java:239) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] handling channelInactive
08:16:25.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:25.275233 10629 connection.cc:573] server connection from 127.10.40.65:53809 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
08:16:25.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:25.275393 10524 connection.cc:573] server connection from 127.10.40.65:35567 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
08:16:25.275 [DEBUG - kudu-nio-1] (Connection.java:709) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] cleaning up while in state READY due to: connection closed
08:16:25.275 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:16:25.276 [DEBUG - Time-limited test] (Connection.java:748) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] connecting to peer
08:16:25.277 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:25.278 [DEBUG - kudu-nio-2] (Connection.java:709) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] cleaning up while in state CONNECTING due to: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:25.278 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:25.287 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:25.287 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983),f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247),f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)[L]]}]
08:16:25.287 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=2, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=11), Traces: [0ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [2ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [2ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [10ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@1958148275(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@5ed95fc8 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5efe7549 -> wakeup thread Time-limited test)) after lookup
08:16:25.288 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] connecting to peer
08:16:25.289 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:25.289 [DEBUG - kudu-nio-3] (Connection.java:709) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] cleaning up while in state CONNECTING due to: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:25.290 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:25.306 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:25.307 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983),f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247),f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)[L]]}]
08:16:25.307 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=4, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=31), Traces: [0ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [2ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [2ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [10ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [12ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [14ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [14ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [30ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@1958148275(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@5ed95fc8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@244359370 after retry RPC@594139244), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5efe7549 -> wakeup thread Time-limited test -> (continuation of Deferred@244359370 after retry RPC@594139244))) after lookup
08:16:25.308 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] connecting to peer
08:16:25.309 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:25.309 [DEBUG - kudu-nio-4] (Connection.java:709) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] cleaning up while in state CONNECTING due to: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:25.309 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:25.347 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:25.347 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983),f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247),f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)[L]]}]
08:16:25.348 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=6, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=72), Traces: [0ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [2ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [2ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [10ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [12ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [14ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [14ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [30ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [31ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [33ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [33ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [67ms] refreshing cache from master, [68ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [70ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@1958148275(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@5ed95fc8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@244359370 after retry RPC@594139244) -> (continuation of Deferred@1807801733 after retry RPC@101175849), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5efe7549 -> wakeup thread Time-limited test -> (continuation of Deferred@244359370 after retry RPC@594139244) -> (continuation of Deferred@1807801733 after retry RPC@101175849))) after lookup
08:16:25.349 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] connecting to peer
08:16:25.350 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:25.350 [DEBUG - kudu-nio-5] (Connection.java:709) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] cleaning up while in state CONNECTING due to: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:25.350 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:25.466 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:25.467 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983),f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247),f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)[L]]}]
08:16:25.468 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=8, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=191), Traces: [0ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [2ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [2ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [10ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [12ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [14ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [14ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [30ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [31ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [33ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [33ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [67ms] refreshing cache from master, [68ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [70ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [72ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [74ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [74ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [187ms] refreshing cache from master, [187ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [190ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@1958148275(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@5ed95fc8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@244359370 after retry RPC@594139244) -> (continuation of Deferred@1807801733 after retry RPC@101175849) -> (continuation of Deferred@276043180 after retry RPC@676989713), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5efe7549 -> wakeup thread Time-limited test -> (continuation of Deferred@244359370 after retry RPC@594139244) -> (continuation of Deferred@1807801733 after retry RPC@101175849) -> (continuation of Deferred@276043180 after retry RPC@676989713))) after lookup
08:16:25.469 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] connecting to peer
08:16:25.470 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:25.470 [DEBUG - kudu-nio-6] (Connection.java:709) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] cleaning up while in state CONNECTING due to: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:25.470 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:25.627 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:25.627 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983),f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247),f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)[L]]}]
08:16:25.628 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=10, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=351), Traces: [0ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [2ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [2ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [10ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [12ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [14ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [14ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [30ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [31ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [33ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [33ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [67ms] refreshing cache from master, [68ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [70ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [72ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [74ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [74ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [187ms] refreshing cache from master, [187ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [190ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [192ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [194ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [194ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [347ms] refreshing cache from master, [347ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [350ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@1958148275(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@5ed95fc8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@244359370 after retry RPC@594139244) -> (continuation of Deferred@1807801733 after retry RPC@101175849) -> (continuation of Deferred@276043180 after retry RPC@676989713) -> (continuation of Deferred@2130030893 after retry RPC@101662561), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5efe7549 -> wakeup thread Time-limited test -> (continuation of Deferred@244359370 after retry RPC@594139244) -> (continuation of Deferred@1807801733 after retry RPC@101175849) -> (continuation of Deferred@276043180 after retry RPC@676989713) -> (continuation of Deferred@2130030893 after retry RPC@101662561))) after lookup
08:16:25.629 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] connecting to peer
08:16:25.630 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:25.631 [DEBUG - kudu-nio-7] (Connection.java:709) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] cleaning up while in state CONNECTING due to: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:25.631 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:26.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:26.275228 11616 raft_consensus.cc:493] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
08:16:26.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:26.275831 11616 raft_consensus.cc:515] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:26.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:26.277726 11616 leader_election.cc:290] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247), f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65:39249)
08:16:26.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:26.280007 11442 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.40.65:39249: connect: Connection refused (error 111)
08:16:26.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:26.281822 11442 leader_election.cc:336] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65:39249): Network error: Client connection negotiation failed: client connection to 127.10.40.65:39249: connect: Connection refused (error 111)
08:16:26.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:26.285689 11049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b328961edc164a1289f259240593e73e" candidate_uuid: "2ea1996689fb4cf18b17a35790fba392" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f0e29bd27cb448589ac37e569ad1a164" is_pre_election: true
08:16:26.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:26.286721 11442 leader_election.cc:304] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ea1996689fb4cf18b17a35790fba392; no voters: f0e29bd27cb448589ac37e569ad1a164, f2a3f45a9e3b4f2dbda7e2613df89453
08:16:26.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:26.287160 11616 raft_consensus.cc:2749] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
08:16:26.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:26.511396 11622 raft_consensus.cc:493] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f2a3f45a9e3b4f2dbda7e2613df89453)
08:16:26.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:26.511591 11622 raft_consensus.cc:515] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:26.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:26.512446 11622 leader_election.cc:290] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67:44983), f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65:39249)
08:16:26.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:26.516124 10966 leader_election.cc:336] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65:39249): Network error: Client connection negotiation failed: client connection to 127.10.40.65:39249: connect: Connection refused (error 111)
08:16:26.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:26.521024 11538 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b328961edc164a1289f259240593e73e" candidate_uuid: "f0e29bd27cb448589ac37e569ad1a164" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "2ea1996689fb4cf18b17a35790fba392" is_pre_election: true
08:16:26.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:26.521452 11538 raft_consensus.cc:2468] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e29bd27cb448589ac37e569ad1a164 in term 1.
08:16:26.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:26.522506 10966 leader_election.cc:304] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ea1996689fb4cf18b17a35790fba392, f0e29bd27cb448589ac37e569ad1a164; no voters: f2a3f45a9e3b4f2dbda7e2613df89453
08:16:26.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:26.522840 11622 raft_consensus.cc:2804] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 2 FOLLOWER]: Leader pre-election won for term 3
08:16:26.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:26.523021 11622 raft_consensus.cc:493] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 2 FOLLOWER]: Starting leader election (detected failure of leader f2a3f45a9e3b4f2dbda7e2613df89453)
08:16:26.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:26.523154 11622 raft_consensus.cc:3060] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 2 FOLLOWER]: Advancing to term 3
08:16:26.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:26.525311 11622 raft_consensus.cc:515] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:26.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:26.526031 11622 leader_election.cc:290] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [CANDIDATE]: Term 3 election: Requested vote from peers 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67:44983), f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65:39249)
08:16:26.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:26.526676 11538 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b328961edc164a1289f259240593e73e" candidate_uuid: "f0e29bd27cb448589ac37e569ad1a164" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "2ea1996689fb4cf18b17a35790fba392"
08:16:26.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:26.526901 11538 raft_consensus.cc:3060] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 1 FOLLOWER]: Advancing to term 3
08:16:26.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:26.528318 10966 leader_election.cc:336] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65:39249): Network error: Client connection negotiation failed: client connection to 127.10.40.65:39249: connect: Connection refused (error 111)
08:16:26.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:26.531212 11538 raft_consensus.cc:2468] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e29bd27cb448589ac37e569ad1a164 in term 3.
08:16:26.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:26.531690 10966 leader_election.cc:304] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ea1996689fb4cf18b17a35790fba392, f0e29bd27cb448589ac37e569ad1a164; no voters: f2a3f45a9e3b4f2dbda7e2613df89453
08:16:26.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:26.532019 11622 raft_consensus.cc:2804] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 3 FOLLOWER]: Leader election won for term 3
08:16:26.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:26.532500 11622 raft_consensus.cc:697] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 3 LEADER]: Becoming Leader. State: Replica: f0e29bd27cb448589ac37e569ad1a164, State: Running, Role: LEADER
08:16:26.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:26.533025 11622 consensus_queue.cc:237] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:26.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:26.536957 10647 catalog_manager.cc:5719] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 reported cstate change: term changed from 2 to 3, leader changed from f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65) to f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66). New cstate: current_term: 3 leader_uuid: "f0e29bd27cb448589ac37e569ad1a164" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } health_report { overall_health: UNKNOWN } } }
08:16:26.667 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:26.668 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983),f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)[L],f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)]}]
08:16:26.668 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=12, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=1392), Traces: [0ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [2ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [2ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [10ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [12ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [14ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [14ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [30ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [31ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [33ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [33ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [67ms] refreshing cache from master, [68ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [70ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [72ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [74ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [74ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [187ms] refreshing cache from master, [187ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [190ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [192ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [194ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [194ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [347ms] refreshing cache from master, [347ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [350ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [352ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [355ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [355ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [1387ms] refreshing cache from master, [1387ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [1391ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@1958148275(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@5ed95fc8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@244359370 after retry RPC@594139244) -> (continuation of Deferred@1807801733 after retry RPC@101175849) -> (continuation of Deferred@276043180 after retry RPC@676989713) -> (continuation of Deferred@2130030893 after retry RPC@101662561) -> (continuation of Deferred@1331145709 after retry RPC@640007461), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5efe7549 -> wakeup thread Time-limited test -> (continuation of Deferred@244359370 after retry RPC@594139244) -> (continuation of Deferred@1807801733 after retry RPC@101175849) -> (continuation of Deferred@276043180 after retry RPC@676989713) -> (continuation of Deferred@2130030893 after retry RPC@101662561) -> (continuation of Deferred@1331145709 after retry RPC@640007461))) after lookup
08:16:26.669 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] connecting to peer
08:16:26.670 [DEBUG - kudu-nio-0] (Connection.java:757) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] Successfully connected to peer
08:16:26.688 [DEBUG - kudu-nio-0] (Negotiator.java:918) Authenticated connection [id: 0x967e732d, L:/127.0.0.1:46422 - R:/127.10.40.66:46247] using TOKEN/null
08:16:26.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:26.694322 11538 raft_consensus.cc:1275] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 3 FOLLOWER]: Refusing update from remote peer f0e29bd27cb448589ac37e569ad1a164: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
08:16:26.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:26.694859 10966 consensus_peers.cc:597] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 -> Peer f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65:39249): Couldn't send request to peer f2a3f45a9e3b4f2dbda7e2613df89453. Status: Network error: Client connection negotiation failed: client connection to 127.10.40.65:39249: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
08:16:26.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:26.695354 11627 consensus_queue.cc:1048] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 4, Time since last communication: 0.000s
08:16:26.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:26.712144 11642 mvcc.cc:204] Tried to move back new op lower bound from 7317596515089625088 to 7317596514443137024. Current Snapshot: MvccSnapshot[applied={T|T < 7317596515089625088}]
08:16:26.737 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.10.40.65:39249
08:16:26.738 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

08:16:26.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:26.739434 10401 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0lHOBl/build/asan/bin/kudu
08:16:26.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task0lHOBl/build/asan/bin/kudu
08:16:26.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/wal
08:16:26.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/data
08:16:26.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:16:26.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:16:26.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:16:26.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:16:26.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:16:26.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:16:26.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/logs
08:16:26.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/data/info.pb
08:16:26.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:16:26.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:16:26.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:16:26.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:16:26.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:16:26.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:16:26.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:16:26.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:16:26.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:16:26.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:16:26.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.40.65:39249
08:16:26.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.40.65
08:16:26.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.40.126:45793,127.10.40.125:46051,127.10.40.124:41773
08:16:26.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=41939
08:16:26.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.40.65
08:16:26.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.40.84:40955
08:16:26.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:16:26.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:16:26.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:16:27.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:27.105026 11652 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:16:27.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:27.105360 11652 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:16:27.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:27.105458 11652 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:16:27.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:27.115180 11652 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:16:27.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:27.115366 11652 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.40.65
08:16:27.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.126745 11652 tablet_server_runner.cc:78] Tablet server non-default flags:
08:16:27.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/logs
08:16:27.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:16:27.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:16:27.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:16:27.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.40.84:40955
08:16:27.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:16:27.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:16:27.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/data
08:16:27.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/wal
08:16:27.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:16:27.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:16:27.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.40.65:39249
08:16:27.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:16:27.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:16:27.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:16:27.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/data/info.pb
08:16:27.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.40.65
08:16:27.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=41939
08:16:27.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.40.126:45793,127.10.40.125:46051,127.10.40.124:41773
08:16:27.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:16:27.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:16:27.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:16:27.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:16:27.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:16:27.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.40.65
08:16:27.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:16:27.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:16:27.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:16:27.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 70c51312f1186a8ae9f46647160b7730123f7601
08:16:27.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:16:27.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 12 Aug 2026 07:43:21 UTC on bdcb31816ec0
08:16:27.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 13302
08:16:27.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
08:16:27.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.129263 11652 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:16:27.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.131488 11652 file_cache.cc:504] Constructed file cache file cache with capacity 419430
08:16:27.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:27.142648 11663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:16:27.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:27.142644 11661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:16:27.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:27.143620 11665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:16:27.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.144430 11652 server_base.cc:1061] running on GCE node
08:16:27.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.145028 11652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:16:27.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.146009 11652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:16:27.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.147233 11652 hybrid_clock.cc:648] HybridClock initialized: now 1786522587147178 us; error 47 us; skew 500 ppm
08:16:27.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.147547 11652 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:16:27.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.149936 11652 webserver.cc:533] Webserver started at http://127.10.40.65:41939/ using document root <none> and password file <none>
08:16:27.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.150983 11652 fs_manager.cc:362] Metadata directory not provided
08:16:27.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.151109 11652 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:16:27.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.156937 11652 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.002s
08:16:27.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.160208 11671 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:16:27.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.161880 11652 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
08:16:27.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.162061 11652 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/data,/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/wal
08:16:27.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "f2a3f45a9e3b4f2dbda7e2613df89453"
08:16:27.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:16:19 on dist-test-slave-gp6n"
08:16:27.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.162992 11652 fs_report.cc:389] FS layout report
08:16:27.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:16:27.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/wal
08:16:27.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/wal
08:16:27.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/data/data
08:16:27.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:16:27.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:16:27.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:16:27.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:16:27.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:16:27.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:16:27.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:16:27.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:16:27.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:16:27.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:16:27.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:16:27.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.189813 11652 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:16:27.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.190704 11652 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:16:27.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.190958 11652 kserver.cc:163] Server-wide thread pool size limit: 3276
08:16:27.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.191887 11652 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
08:16:27.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.194164 11681 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
08:16:27.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.200243 11652 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
08:16:27.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.200337 11652 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.007s	user 0.001s	sys 0.000s
08:16:27.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.200436 11652 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
08:16:27.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.203548 11652 ts_tablet_manager.cc:616] Registered 1 tablets
08:16:27.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.203648 11652 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.002s	sys 0.000s
08:16:27.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.204272 11681 tablet_bootstrap.cc:492] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453: Bootstrap starting.
08:16:27.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.230197 11681 log.cc:826] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453: Log is configured to *not* fsync() on all Append() calls
08:16:27.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.256325 11681 tablet_bootstrap.cc:492] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=6 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
08:16:27.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.256995 11681 tablet_bootstrap.cc:492] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453: Bootstrap complete.
08:16:27.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.258455 11681 ts_tablet_manager.cc:1403] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453: Time spent bootstrapping tablet: real 0.055s	user 0.038s	sys 0.012s
08:16:27.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.259754 11652 rpc_server.cc:307] RPC server started. Bound to: 127.10.40.65:39249
08:16:27.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.260048 11811 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.40.65:39249 every 8 connection(s)
08:16:27.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.261343 11652 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/data/info.pb
08:16:27.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.264672 10401 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0lHOBl/build/asan/bin/kudu as pid 11652
08:16:27.265 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:16:27.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.272408 11681 raft_consensus.cc:359] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:27.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.274363 11681 raft_consensus.cc:740] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2a3f45a9e3b4f2dbda7e2613df89453, State: Initialized, Role: FOLLOWER
08:16:27.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.275664 11681 consensus_queue.cc:260] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:27.294 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 9 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644, tablet=b328961edc164a1289f259240593e73e, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:16:27.295 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.10.40.66:46247
08:16:27.296 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

08:16:27.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.296381 10401 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-task0lHOBl/build/asan/bin/kudu pid 10948
08:16:27.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.296711 11681 ts_tablet_manager.cc:1434] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453: Time spent starting tablet: real 0.038s	user 0.008s	sys 0.015s
08:16:27.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.308120 11099 generic_service.cc:196] Checking for leaks (request via RPC)
08:16:27.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.308640 11815 heartbeater.cc:344] Connected to a master server at 127.10.40.125:46051
08:16:27.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.309091 11815 heartbeater.cc:461] Registering TS with master...
08:16:27.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.309612 11814 heartbeater.cc:344] Connected to a master server at 127.10.40.126:45793
08:16:27.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.309832 11814 heartbeater.cc:461] Registering TS with master...
08:16:27.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.310393 11815 heartbeater.cc:507] Master 127.10.40.125:46051 requested a full tablet report, sending...
08:16:27.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.310631 11814 heartbeater.cc:507] Master 127.10.40.126:45793 requested a full tablet report, sending...
08:16:27.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.312098 11813 heartbeater.cc:344] Connected to a master server at 127.10.40.124:41773
08:16:27.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.312230 11813 heartbeater.cc:461] Registering TS with master...
08:16:27.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.312821 11813 heartbeater.cc:507] Master 127.10.40.124:41773 requested a full tablet report, sending...
08:16:27.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.314105 10451 ts_manager.cc:194] Re-registered known tserver with Master: f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65:39249)
08:16:27.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.314227 10550 ts_manager.cc:194] Re-registered known tserver with Master: f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65:39249)
08:16:27.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.314270 10647 ts_manager.cc:194] Re-registered known tserver with Master: f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65:39249)
08:16:27.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.315794 10647 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.10.40.65:36545
08:16:27.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.318248 11813 heartbeater.cc:499] Master 127.10.40.124:41773 was elected leader, sending a full tablet report...
08:16:27.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:27.418288 10401 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0lHOBl/build/asan/bin/kudu with pid 10948
08:16:27.446 [DEBUG - kudu-nio-0] (Connection.java:239) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] handling channelInactive
08:16:27.446 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:16:27.446 [DEBUG - kudu-nio-0] (Connection.java:709) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] cleaning up while in state READY due to: connection closed
08:16:27.447 [DEBUG - Time-limited test] (Connection.java:748) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] connecting to peer
08:16:27.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:27.447001 11830 negotiation.cc:336] Failed RPC negotiation. Trace:
08:16:27.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:16:27.423514 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.10.40.66:56501 (local address 127.10.40.65:39249)
08:16:27.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:16:27.424326 (+   812us) server_negotiation.cc:207] Beginning negotiation
08:16:27.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:16:27.424335 (+     9us) server_negotiation.cc:400] Waiting for connection header
08:16:27.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0812 08:16:27.445450 (+ 21115us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.40.66:56501: BlockingRecv error: recv got EOF from 127.10.40.66:56501 (error 108)
08:16:27.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":2987,"threads_started":1}
08:16:27.448 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:27.448 [DEBUG - kudu-nio-1] (Connection.java:709) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] cleaning up while in state CONNECTING due to: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:27.449 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:27.468 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:27.468 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983),f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)[L],f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)]}]
08:16:27.468 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=2, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=22), Traces: [0ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [2ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [3ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [17ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [21ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@1682602503(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@164aea22 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5504dc10 -> wakeup thread Time-limited test)) after lookup
08:16:27.469 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] connecting to peer
08:16:27.470 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:27.471 [DEBUG - kudu-nio-2] (Connection.java:709) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] cleaning up while in state CONNECTING due to: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:27.471 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:27.487 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:27.487 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983),f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)[L],f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)]}]
08:16:27.488 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=4, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=41), Traces: [0ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [2ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [3ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [17ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [21ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [23ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [25ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [25ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [41ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@1682602503(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@164aea22 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1164189487 after retry RPC@913551311), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5504dc10 -> wakeup thread Time-limited test -> (continuation of Deferred@1164189487 after retry RPC@913551311))) after lookup
08:16:27.488 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] connecting to peer
08:16:27.489 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:27.489 [DEBUG - kudu-nio-3] (Connection.java:709) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] cleaning up while in state CONNECTING due to: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:27.490 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:27.527 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:27.527 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983),f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)[L],f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)]}]
08:16:27.527 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=6, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=81), Traces: [0ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [2ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [3ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [17ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [21ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [23ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [25ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [25ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [41ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [42ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [43ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [44ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [80ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@1682602503(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@164aea22 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1164189487 after retry RPC@913551311) -> (continuation of Deferred@177104767 after retry RPC@564702262), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5504dc10 -> wakeup thread Time-limited test -> (continuation of Deferred@1164189487 after retry RPC@913551311) -> (continuation of Deferred@177104767 after retry RPC@564702262))) after lookup
08:16:27.529 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] connecting to peer
08:16:27.530 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:27.530 [DEBUG - kudu-nio-4] (Connection.java:709) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] cleaning up while in state CONNECTING due to: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:27.530 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:27.627 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:27.628 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983),f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)[L],f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)]}]
08:16:27.628 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=8, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=181), Traces: [0ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [2ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [3ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [17ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [21ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [23ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [25ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [25ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [41ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [42ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [43ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [44ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [80ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [82ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [84ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [85ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [178ms] refreshing cache from master, [178ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [181ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@1682602503(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@164aea22 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1164189487 after retry RPC@913551311) -> (continuation of Deferred@177104767 after retry RPC@564702262) -> (continuation of Deferred@1582869385 after retry RPC@980776296), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5504dc10 -> wakeup thread Time-limited test -> (continuation of Deferred@1164189487 after retry RPC@913551311) -> (continuation of Deferred@177104767 after retry RPC@564702262) -> (continuation of Deferred@1582869385 after retry RPC@980776296))) after lookup
08:16:27.629 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] connecting to peer
08:16:27.630 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:27.630 [DEBUG - kudu-nio-5] (Connection.java:709) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] cleaning up while in state CONNECTING due to: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:27.630 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:27.706 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:27.706 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983),f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)[L],f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)]}]
08:16:27.707 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=10, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=260), Traces: [0ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [2ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [3ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [17ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [21ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [23ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [25ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [25ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [41ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [42ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [43ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [44ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [80ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [82ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [84ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [85ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [178ms] refreshing cache from master, [178ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [181ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [182ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [184ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [184ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [257ms] refreshing cache from master, [257ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [260ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@1682602503(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@164aea22 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1164189487 after retry RPC@913551311) -> (continuation of Deferred@177104767 after retry RPC@564702262) -> (continuation of Deferred@1582869385 after retry RPC@980776296) -> (continuation of Deferred@603652538 after retry RPC@842329752), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5504dc10 -> wakeup thread Time-limited test -> (continuation of Deferred@1164189487 after retry RPC@913551311) -> (continuation of Deferred@177104767 after retry RPC@564702262) -> (continuation of Deferred@1582869385 after retry RPC@980776296) -> (continuation of Deferred@603652538 after retry RPC@842329752))) after lookup
08:16:27.707 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] connecting to peer
08:16:27.708 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:27.709 [DEBUG - kudu-nio-6] (Connection.java:709) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] cleaning up while in state CONNECTING due to: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:27.709 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:28.227 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:28.227 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983),f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)[L],f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)]}]
08:16:28.227 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=12, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=781), Traces: [0ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [2ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [3ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [17ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [21ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [23ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [25ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [25ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [41ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [42ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [43ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [44ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [80ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [82ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [84ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [85ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [178ms] refreshing cache from master, [178ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [181ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [182ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [184ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [184ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [257ms] refreshing cache from master, [257ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [260ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [261ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [263ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [263ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [777ms] refreshing cache from master, [777ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [780ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@1682602503(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@164aea22 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1164189487 after retry RPC@913551311) -> (continuation of Deferred@177104767 after retry RPC@564702262) -> (continuation of Deferred@1582869385 after retry RPC@980776296) -> (continuation of Deferred@603652538 after retry RPC@842329752) -> (continuation of Deferred@290192620 after retry RPC@77500531), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5504dc10 -> wakeup thread Time-limited test -> (continuation of Deferred@1164189487 after retry RPC@913551311) -> (continuation of Deferred@177104767 after retry RPC@564702262) -> (continuation of Deferred@1582869385 after retry RPC@980776296) -> (continuation of Deferred@603652538 after retry RPC@842329752) -> (continuation of Deferred@290192620 after retry RPC@77500531))) after lookup
08:16:28.228 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] connecting to peer
08:16:28.229 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:28.230 [DEBUG - kudu-nio-7] (Connection.java:709) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] cleaning up while in state CONNECTING due to: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:28.230 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:28.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:28.662981 11834 raft_consensus.cc:493] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
08:16:28.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:28.663290 11834 raft_consensus.cc:515] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:28.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:28.664915 11834 leader_election.cc:290] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247), 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67:44983)
08:16:28.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:28.666293 11672 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.40.66:46247: connect: Connection refused (error 111)
08:16:28.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:28.668978 11672 leader_election.cc:336] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247): Network error: Client connection negotiation failed: client connection to 127.10.40.66:46247: connect: Connection refused (error 111)
08:16:28.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:28.672325 11538 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b328961edc164a1289f259240593e73e" candidate_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "2ea1996689fb4cf18b17a35790fba392" is_pre_election: true
08:16:28.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:28.673216 11672 leader_election.cc:304] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2a3f45a9e3b4f2dbda7e2613df89453; no voters: 2ea1996689fb4cf18b17a35790fba392, f0e29bd27cb448589ac37e569ad1a164
08:16:28.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:28.673626 11834 raft_consensus.cc:2749] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
08:16:28.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:28.754971 11840 raft_consensus.cc:493] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f0e29bd27cb448589ac37e569ad1a164)
08:16:28.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:28.755177 11840 raft_consensus.cc:515] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:28.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:28.755925 11840 leader_election.cc:290] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247), f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65:39249)
08:16:28.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:28.758755 11442 leader_election.cc:336] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247): Network error: Client connection negotiation failed: client connection to 127.10.40.66:46247: connect: Connection refused (error 111)
08:16:28.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:28.764688 11756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b328961edc164a1289f259240593e73e" candidate_uuid: "2ea1996689fb4cf18b17a35790fba392" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" is_pre_election: true
08:16:28.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:28.765107 11756 raft_consensus.cc:2468] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ea1996689fb4cf18b17a35790fba392 in term 2.
08:16:28.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:28.766062 11442 leader_election.cc:304] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ea1996689fb4cf18b17a35790fba392, f2a3f45a9e3b4f2dbda7e2613df89453; no voters: f0e29bd27cb448589ac37e569ad1a164
08:16:28.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:28.766472 11840 raft_consensus.cc:2804] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 3 FOLLOWER]: Leader pre-election won for term 4
08:16:28.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:28.766610 11840 raft_consensus.cc:493] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 3 FOLLOWER]: Starting leader election (detected failure of leader f0e29bd27cb448589ac37e569ad1a164)
08:16:28.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:28.766682 11840 raft_consensus.cc:3060] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 3 FOLLOWER]: Advancing to term 4
08:16:28.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:28.769083 11840 raft_consensus.cc:515] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:28.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:28.769665 11840 leader_election.cc:290] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [CANDIDATE]: Term 4 election: Requested vote from peers f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247), f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65:39249)
08:16:28.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:28.770776 11756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b328961edc164a1289f259240593e73e" candidate_uuid: "2ea1996689fb4cf18b17a35790fba392" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453"
08:16:28.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:28.771056 11756 raft_consensus.cc:3060] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 2 FOLLOWER]: Advancing to term 4
08:16:28.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:28.771821 11442 leader_election.cc:336] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247): Network error: Client connection negotiation failed: client connection to 127.10.40.66:46247: connect: Connection refused (error 111)
08:16:28.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:28.775314 11756 raft_consensus.cc:2468] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ea1996689fb4cf18b17a35790fba392 in term 4.
08:16:28.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:28.775815 11442 leader_election.cc:304] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ea1996689fb4cf18b17a35790fba392, f2a3f45a9e3b4f2dbda7e2613df89453; no voters: f0e29bd27cb448589ac37e569ad1a164
08:16:28.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:28.776135 11840 raft_consensus.cc:2804] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 4 FOLLOWER]: Leader election won for term 4
08:16:28.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:28.776662 11840 raft_consensus.cc:697] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 4 LEADER]: Becoming Leader. State: Replica: 2ea1996689fb4cf18b17a35790fba392, State: Running, Role: LEADER
08:16:28.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:28.777221 11840 consensus_queue.cc:237] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:28.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:28.781205 10647 catalog_manager.cc:5719] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 reported cstate change: term changed from 3 to 4, leader changed from f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66) to 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67). New cstate: current_term: 4 leader_uuid: "2ea1996689fb4cf18b17a35790fba392" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } health_report { overall_health: UNKNOWN } } }
08:16:29.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:29.337103 11442 consensus_peers.cc:597] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 -> Peer f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247): Couldn't send request to peer f0e29bd27cb448589ac37e569ad1a164. Status: Network error: Client connection negotiation failed: client connection to 127.10.40.66:46247: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
08:16:29.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:29.365381 11756 raft_consensus.cc:1275] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 4 FOLLOWER]: Refusing update from remote peer 2ea1996689fb4cf18b17a35790fba392: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 4 index: 13. (index mismatch)
08:16:29.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:29.366319 11863 consensus_queue.cc:1048] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 8, Time since last communication: 0.000s
08:16:31.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:31.656637 11442 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.40.66:46247: connect: Connection refused (error 111) [suppressed 7 similar messages]
08:16:31.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:31.657759 11442 consensus_peers.cc:597] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 -> Peer f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247): Couldn't send request to peer f0e29bd27cb448589ac37e569ad1a164. Status: Network error: Client connection negotiation failed: client connection to 127.10.40.66:46247: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
08:16:31.686 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:31.687 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)[L],f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247),f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)]}]
08:16:31.687 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=14, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=4240), Traces: [0ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [2ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [3ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [17ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [21ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [23ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [25ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [25ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [41ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [42ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [43ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [44ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [80ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [82ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [84ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [85ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [178ms] refreshing cache from master, [178ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [181ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [182ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [184ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [184ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [257ms] refreshing cache from master, [257ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [260ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [261ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [263ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [263ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [777ms] refreshing cache from master, [777ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [780ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [782ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [784ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [784ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [4237ms] refreshing cache from master, [4237ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [4240ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@1682602503(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@164aea22 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1164189487 after retry RPC@913551311) -> (continuation of Deferred@177104767 after retry RPC@564702262) -> (continuation of Deferred@1582869385 after retry RPC@980776296) -> (continuation of Deferred@603652538 after retry RPC@842329752) -> (continuation of Deferred@290192620 after retry RPC@77500531) -> (continuation of Deferred@2056444099 after retry RPC@184537834), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5504dc10 -> wakeup thread Time-limited test -> (continuation of Deferred@1164189487 after retry RPC@913551311) -> (continuation of Deferred@177104767 after retry RPC@564702262) -> (continuation of Deferred@1582869385 after retry RPC@980776296) -> (continuation of Deferred@603652538 after retry RPC@842329752) -> (continuation of Deferred@290192620 after retry RPC@77500531) -> (continuation of Deferred@2056444099 after retry RPC@184537834))) after lookup
08:16:31.688 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] connecting to peer
08:16:31.689 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] Successfully connected to peer
08:16:31.706 [DEBUG - kudu-nio-0] (Negotiator.java:918) Authenticated connection [id: 0xfaf411b6, L:/127.0.0.1:48742 - R:/127.10.40.67:44983] using TOKEN/null
08:16:31.740 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.10.40.66:46247
08:16:31.741 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

08:16:31.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:31.741842 10401 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0lHOBl/build/asan/bin/kudu
08:16:31.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task0lHOBl/build/asan/bin/kudu
08:16:31.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/wal
08:16:31.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/data
08:16:31.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:16:31.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:16:31.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:16:31.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:16:31.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:16:31.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:16:31.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/logs
08:16:31.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/data/info.pb
08:16:31.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:16:31.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:16:31.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:16:31.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:16:31.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:16:31.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:16:31.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:16:31.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:16:31.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:16:31.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:16:31.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.40.66:46247
08:16:31.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.40.66
08:16:31.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.40.126:45793,127.10.40.125:46051,127.10.40.124:41773
08:16:31.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=33605
08:16:31.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.40.66
08:16:31.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.40.84:40955
08:16:31.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:16:31.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:16:31.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:16:32.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:32.101460 11898 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:16:32.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:32.101794 11898 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:16:32.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:32.101876 11898 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:16:32.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:32.110458 11898 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:16:32.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:32.110637 11898 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.40.66
08:16:32.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.121474 11898 tablet_server_runner.cc:78] Tablet server non-default flags:
08:16:32.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/logs
08:16:32.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:16:32.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:16:32.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:16:32.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.40.84:40955
08:16:32.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:16:32.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:16:32.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/data
08:16:32.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/wal
08:16:32.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:16:32.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:16:32.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.40.66:46247
08:16:32.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:16:32.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:16:32.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:16:32.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/data/info.pb
08:16:32.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.40.66
08:16:32.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=33605
08:16:32.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.40.126:45793,127.10.40.125:46051,127.10.40.124:41773
08:16:32.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:16:32.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:16:32.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:16:32.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:16:32.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:16:32.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.40.66
08:16:32.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:16:32.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:16:32.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:16:32.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 70c51312f1186a8ae9f46647160b7730123f7601
08:16:32.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:16:32.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 12 Aug 2026 07:43:21 UTC on bdcb31816ec0
08:16:32.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 13302
08:16:32.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
08:16:32.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.123679 11898 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:16:32.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.125574 11898 file_cache.cc:504] Constructed file cache file cache with capacity 419430
08:16:32.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:32.136065 11907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:16:32.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:32.136546 11905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:16:32.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:32.136600 11910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:16:32.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.138192 11898 server_base.cc:1061] running on GCE node
08:16:32.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.138954 11898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:16:32.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.140058 11898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:16:32.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.141368 11898 hybrid_clock.cc:648] HybridClock initialized: now 1786522592141293 us; error 56 us; skew 500 ppm
08:16:32.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.141785 11898 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:16:32.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.144326 11898 webserver.cc:533] Webserver started at http://127.10.40.66:33605/ using document root <none> and password file <none>
08:16:32.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.145200 11898 fs_manager.cc:362] Metadata directory not provided
08:16:32.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.145380 11898 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:16:32.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.151773 11898 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.000s	sys 0.004s
08:16:32.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.154943 11918 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:16:32.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.156661 11898 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.004s
08:16:32.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.156895 11898 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/data,/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/wal
08:16:32.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "f0e29bd27cb448589ac37e569ad1a164"
08:16:32.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:16:19 on dist-test-slave-gp6n"
08:16:32.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.157768 11898 fs_report.cc:389] FS layout report
08:16:32.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:16:32.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/wal
08:16:32.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/wal
08:16:32.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/data/data
08:16:32.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:16:32.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:16:32.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:16:32.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:16:32.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:16:32.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:16:32.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:16:32.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:16:32.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:16:32.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:16:32.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:16:32.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.174968 11898 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:16:32.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.176066 11898 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:16:32.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.176476 11898 kserver.cc:163] Server-wide thread pool size limit: 3276
08:16:32.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.177685 11898 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
08:16:32.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.180402 11926 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
08:16:32.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.186668 11898 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
08:16:32.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.186813 11898 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.007s	user 0.001s	sys 0.000s
08:16:32.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.186988 11898 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
08:16:32.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.190461 11898 ts_tablet_manager.cc:616] Registered 1 tablets
08:16:32.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.190620 11898 ts_tablet_manager.cc:595] Time spent register tablets: real 0.004s	user 0.000s	sys 0.003s
08:16:32.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.191023 11926 tablet_bootstrap.cc:492] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164: Bootstrap starting.
08:16:32.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.216768 11926 log.cc:826] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164: Log is configured to *not* fsync() on all Append() calls
08:16:32.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.241360 11926 tablet_bootstrap.cc:492] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=9 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
08:16:32.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.241969 11926 tablet_bootstrap.cc:492] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164: Bootstrap complete.
08:16:32.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.243597 11926 ts_tablet_manager.cc:1403] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164: Time spent bootstrapping tablet: real 0.053s	user 0.027s	sys 0.023s
08:16:32.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.249687 11926 raft_consensus.cc:359] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:32.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.250777 11926 raft_consensus.cc:740] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e29bd27cb448589ac37e569ad1a164, State: Initialized, Role: FOLLOWER
08:16:32.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.251778 11926 consensus_queue.cc:260] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:32.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.253638 11926 ts_tablet_manager.cc:1434] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164: Time spent starting tablet: real 0.010s	user 0.009s	sys 0.001s
08:16:32.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.262820 11898 rpc_server.cc:307] RPC server started. Bound to: 127.10.40.66:46247
08:16:32.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.262849 12061 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.40.66:46247 every 8 connection(s)
08:16:32.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.264756 11898 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/data/info.pb
08:16:32.270 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:16:32.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.270154 10401 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0lHOBl/build/asan/bin/kudu as pid 11898
08:16:32.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.295612 12007 raft_consensus.cc:3060] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 3 FOLLOWER]: Advancing to term 4
08:16:32.315 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 12 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644, tablet=b328961edc164a1289f259240593e73e, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:16:32.323 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.10.40.67:44983
08:16:32.323 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

08:16:32.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.324132 10401 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-task0lHOBl/build/asan/bin/kudu pid 11420
08:16:32.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.328234 12064 heartbeater.cc:344] Connected to a master server at 127.10.40.124:41773
08:16:32.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.328686 12064 heartbeater.cc:461] Registering TS with master...
08:16:32.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.329890 12064 heartbeater.cc:507] Master 127.10.40.124:41773 requested a full tablet report, sending...
08:16:32.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.344272 12074 mvcc.cc:204] Tried to move back new op lower bound from 7317596535648886784 to 7317596523635585024. Current Snapshot: MvccSnapshot[applied={T|T < 7317596535648886784}]
08:16:32.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.347374 12071 heartbeater.cc:344] Connected to a master server at 127.10.40.125:46051
08:16:32.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.347554 12071 heartbeater.cc:461] Registering TS with master...
08:16:32.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.348146 12071 heartbeater.cc:507] Master 127.10.40.125:46051 requested a full tablet report, sending...
08:16:32.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.348577 12067 heartbeater.cc:344] Connected to a master server at 127.10.40.126:45793
08:16:32.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.348775 12067 heartbeater.cc:461] Registering TS with master...
08:16:32.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.349500 12067 heartbeater.cc:507] Master 127.10.40.126:45793 requested a full tablet report, sending...
08:16:32.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.360128 11590 generic_service.cc:196] Checking for leaks (request via RPC)
08:16:32.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.362084 10451 ts_manager.cc:194] Re-registered known tserver with Master: f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247)
08:16:32.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.362138 10648 ts_manager.cc:194] Re-registered known tserver with Master: f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247)
08:16:32.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.362341 10550 ts_manager.cc:194] Re-registered known tserver with Master: f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247)
08:16:32.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.363757 10648 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.10.40.66:48333
08:16:32.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.367264 12064 heartbeater.cc:499] Master 127.10.40.124:41773 was elected leader, sending a full tablet report...
08:16:32.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:32.469713 10401 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0lHOBl/build/asan/bin/kudu with pid 11420
08:16:32.497 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:16:32.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:32.497871 10633 connection.cc:573] server connection from 127.10.40.67:44145 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
08:16:32.498 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] handling channelInactive
08:16:32.498 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] cleaning up while in state READY due to: connection closed
08:16:32.504 [DEBUG - pool-2-thread-1] (Connection.java:748) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] connecting to peer
08:16:32.505 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:32.505 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] cleaning up while in state CONNECTING due to: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:32.505 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:32.526 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:32.526 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)[L],f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247),f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)]}]
08:16:32.527 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=3, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=28), Traces: [0ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [0ms] delaying RPC due to: Illegal state: connection is terminated, [5ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [7ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [7ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [28ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@765837053(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@909751f -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4a28517d -> wakeup thread Time-limited test)) after lookup
08:16:32.527 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] connecting to peer
08:16:32.528 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:32.528 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] cleaning up while in state CONNECTING due to: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:32.529 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:32.567 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:32.567 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)[L],f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247),f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)]}]
08:16:32.567 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=5, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=69), Traces: [0ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [0ms] delaying RPC due to: Illegal state: connection is terminated, [5ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [7ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [7ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [28ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [29ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [30ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [31ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [65ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [68ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@765837053(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@909751f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@337447718 after retry RPC@473305953), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4a28517d -> wakeup thread Time-limited test -> (continuation of Deferred@337447718 after retry RPC@473305953))) after lookup
08:16:32.569 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] connecting to peer
08:16:32.570 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:32.570 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] cleaning up while in state CONNECTING due to: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:32.570 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:32.607 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:32.607 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)[L],f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247),f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)]}]
08:16:32.607 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=7, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=109), Traces: [0ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [0ms] delaying RPC due to: Illegal state: connection is terminated, [5ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [7ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [7ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [28ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [29ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [30ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [31ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [65ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [68ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [70ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [72ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [72ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [105ms] refreshing cache from master, [105ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [108ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@765837053(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@909751f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@337447718 after retry RPC@473305953) -> (continuation of Deferred@1891611660 after retry RPC@1547574369), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4a28517d -> wakeup thread Time-limited test -> (continuation of Deferred@337447718 after retry RPC@473305953) -> (continuation of Deferred@1891611660 after retry RPC@1547574369))) after lookup
08:16:32.608 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] connecting to peer
08:16:32.609 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:32.610 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] cleaning up while in state CONNECTING due to: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:32.610 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:32.807 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:32.807 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)[L],f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247),f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)]}]
08:16:32.807 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=9, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=309), Traces: [0ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [0ms] delaying RPC due to: Illegal state: connection is terminated, [5ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [7ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [7ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [28ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [29ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [30ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [31ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [65ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [68ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [70ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [72ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [72ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [105ms] refreshing cache from master, [105ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [108ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [110ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [112ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [112ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [305ms] refreshing cache from master, [305ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [308ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@765837053(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@909751f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@337447718 after retry RPC@473305953) -> (continuation of Deferred@1891611660 after retry RPC@1547574369) -> (continuation of Deferred@1703273135 after retry RPC@1232349034), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4a28517d -> wakeup thread Time-limited test -> (continuation of Deferred@337447718 after retry RPC@473305953) -> (continuation of Deferred@1891611660 after retry RPC@1547574369) -> (continuation of Deferred@1703273135 after retry RPC@1232349034))) after lookup
08:16:32.808 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] connecting to peer
08:16:32.810 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:32.810 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] cleaning up while in state CONNECTING due to: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:32.810 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:33.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:33.624246 12085 raft_consensus.cc:493] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 2ea1996689fb4cf18b17a35790fba392)
08:16:33.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:33.624543 12085 raft_consensus.cc:515] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:33.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:33.625595 12085 leader_election.cc:290] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247), 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67:44983)
08:16:33.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:33.631850 11672 leader_election.cc:336] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67:44983): Network error: Client connection negotiation failed: client connection to 127.10.40.67:44983: connect: Connection refused (error 111)
08:16:33.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:33.633307 12007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b328961edc164a1289f259240593e73e" candidate_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "f0e29bd27cb448589ac37e569ad1a164" is_pre_election: true
08:16:33.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:33.634337 11672 leader_election.cc:304] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2a3f45a9e3b4f2dbda7e2613df89453; no voters: 2ea1996689fb4cf18b17a35790fba392, f0e29bd27cb448589ac37e569ad1a164
08:16:33.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:33.634688 12085 raft_consensus.cc:2749] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
08:16:33.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:33.804571 12091 raft_consensus.cc:493] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 2ea1996689fb4cf18b17a35790fba392)
08:16:33.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:33.804903 12091 raft_consensus.cc:515] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:33.807 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:33.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:33.807051 12091 leader_election.cc:290] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67:44983), f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65:39249)
08:16:33.807 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)[L],f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247),f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)]}]
08:16:33.808 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=11, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=1309), Traces: [0ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [0ms] delaying RPC due to: Illegal state: connection is terminated, [5ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [7ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [7ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [28ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [29ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [30ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [31ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [65ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [68ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [70ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [72ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [72ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [105ms] refreshing cache from master, [105ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [108ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [110ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [112ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [112ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [305ms] refreshing cache from master, [305ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [308ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [310ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [312ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [312ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [1305ms] refreshing cache from master, [1305ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [1308ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@765837053(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@909751f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@337447718 after retry RPC@473305953) -> (continuation of Deferred@1891611660 after retry RPC@1547574369) -> (continuation of Deferred@1703273135 after retry RPC@1232349034) -> (continuation of Deferred@958527906 after retry RPC@1367787163), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4a28517d -> wakeup thread Time-limited test -> (continuation of Deferred@337447718 after retry RPC@473305953) -> (continuation of Deferred@1891611660 after retry RPC@1547574369) -> (continuation of Deferred@1703273135 after retry RPC@1232349034) -> (continuation of Deferred@958527906 after retry RPC@1367787163))) after lookup
08:16:33.809 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] connecting to peer
08:16:33.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:33.809235 11919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.40.67:44983: connect: Connection refused (error 111)
08:16:33.812 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:33.813 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] cleaning up while in state CONNECTING due to: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:33.813 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:33.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:33.813565 11919 leader_election.cc:336] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67:44983): Network error: Client connection negotiation failed: client connection to 127.10.40.67:44983: connect: Connection refused (error 111)
08:16:33.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:33.818014 11756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b328961edc164a1289f259240593e73e" candidate_uuid: "f0e29bd27cb448589ac37e569ad1a164" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" is_pre_election: true
08:16:33.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:33.818274 11756 raft_consensus.cc:2468] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e29bd27cb448589ac37e569ad1a164 in term 4.
08:16:33.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:33.818893 11919 leader_election.cc:304] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f0e29bd27cb448589ac37e569ad1a164, f2a3f45a9e3b4f2dbda7e2613df89453; no voters: 2ea1996689fb4cf18b17a35790fba392
08:16:33.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:33.819392 12091 raft_consensus.cc:2804] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 4 FOLLOWER]: Leader pre-election won for term 5
08:16:33.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:33.819515 12091 raft_consensus.cc:493] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 2ea1996689fb4cf18b17a35790fba392)
08:16:33.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:33.819594 12091 raft_consensus.cc:3060] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 4 FOLLOWER]: Advancing to term 5
08:16:33.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:33.821843 12091 raft_consensus.cc:515] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:33.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:33.822382 12091 leader_election.cc:290] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [CANDIDATE]: Term 5 election: Requested vote from peers 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67:44983), f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65:39249)
08:16:33.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:33.823544 11756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b328961edc164a1289f259240593e73e" candidate_uuid: "f0e29bd27cb448589ac37e569ad1a164" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453"
08:16:33.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:33.823797 11756 raft_consensus.cc:3060] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 4 FOLLOWER]: Advancing to term 5
08:16:33.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:33.824244 11919 leader_election.cc:336] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67:44983): Network error: Client connection negotiation failed: client connection to 127.10.40.67:44983: connect: Connection refused (error 111)
08:16:33.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:33.826031 11756 raft_consensus.cc:2468] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e29bd27cb448589ac37e569ad1a164 in term 5.
08:16:33.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:33.826597 11919 leader_election.cc:304] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f0e29bd27cb448589ac37e569ad1a164, f2a3f45a9e3b4f2dbda7e2613df89453; no voters: 2ea1996689fb4cf18b17a35790fba392
08:16:33.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:33.826963 12091 raft_consensus.cc:2804] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 5 FOLLOWER]: Leader election won for term 5
08:16:33.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:33.827455 12091 raft_consensus.cc:697] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 5 LEADER]: Becoming Leader. State: Replica: f0e29bd27cb448589ac37e569ad1a164, State: Running, Role: LEADER
08:16:33.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:33.828069 12091 consensus_queue.cc:237] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 4.16, Last appended by leader: 16, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:33.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:33.832224 10648 catalog_manager.cc:5719] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 reported cstate change: term changed from 4 to 5, leader changed from 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67) to f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66). New cstate: current_term: 5 leader_uuid: "f0e29bd27cb448589ac37e569ad1a164" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } health_report { overall_health: UNKNOWN } } }
08:16:34.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:34.366283 11756 raft_consensus.cc:1275] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 5 FOLLOWER]: Refusing update from remote peer f0e29bd27cb448589ac37e569ad1a164: Log matching property violated. Preceding OpId in replica: term: 4 index: 16. Preceding OpId from leader: term: 5 index: 17. (index mismatch)
08:16:34.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:34.367285 12114 consensus_queue.cc:1048] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
08:16:34.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:34.380863 11919 consensus_peers.cc:597] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 -> Peer 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67:44983): Couldn't send request to peer 2ea1996689fb4cf18b17a35790fba392. Status: Network error: Client connection negotiation failed: client connection to 127.10.40.67:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
08:16:36.586 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:36.586 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983),f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)[L],f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)]}]
08:16:36.587 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=13, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=4088), Traces: [0ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [0ms] delaying RPC due to: Illegal state: connection is terminated, [5ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [7ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [7ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [28ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [29ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [30ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [31ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [65ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [68ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [70ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [72ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [72ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [105ms] refreshing cache from master, [105ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [108ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [110ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [112ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [112ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [305ms] refreshing cache from master, [305ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [308ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [310ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [312ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [312ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [1305ms] refreshing cache from master, [1305ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [1308ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [1310ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [1315ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [1315ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [4085ms] refreshing cache from master, [4085ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [4088ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@765837053(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@909751f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@337447718 after retry RPC@473305953) -> (continuation of Deferred@1891611660 after retry RPC@1547574369) -> (continuation of Deferred@1703273135 after retry RPC@1232349034) -> (continuation of Deferred@958527906 after retry RPC@1367787163) -> (continuation of Deferred@367767391 after retry RPC@735085807), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4a28517d -> wakeup thread Time-limited test -> (continuation of Deferred@337447718 after retry RPC@473305953) -> (continuation of Deferred@1891611660 after retry RPC@1547574369) -> (continuation of Deferred@1703273135 after retry RPC@1232349034) -> (continuation of Deferred@958527906 after retry RPC@1367787163) -> (continuation of Deferred@367767391 after retry RPC@735085807))) after lookup
08:16:36.588 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] connecting to peer
08:16:36.589 [DEBUG - kudu-nio-7] (Connection.java:757) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] Successfully connected to peer
08:16:36.622 [DEBUG - kudu-nio-7] (Negotiator.java:918) Authenticated connection [id: 0x9b3998af, L:/127.0.0.1:37602 - R:/127.10.40.66:46247] using TOKEN/null
08:16:36.651 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.10.40.67:44983
08:16:36.651 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

08:16:36.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:36.652109 10401 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0lHOBl/build/asan/bin/kudu
08:16:36.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task0lHOBl/build/asan/bin/kudu
08:16:36.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/wal
08:16:36.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/data
08:16:36.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:16:36.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:16:36.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:16:36.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:16:36.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:16:36.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:16:36.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/logs
08:16:36.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/data/info.pb
08:16:36.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:16:36.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:16:36.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:16:36.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:16:36.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:16:36.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:16:36.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:16:36.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:16:36.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:16:36.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:16:36.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.40.67:44983
08:16:36.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.40.67
08:16:36.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.40.126:45793,127.10.40.125:46051,127.10.40.124:41773
08:16:36.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45387
08:16:36.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.40.67
08:16:36.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.40.84:40955
08:16:36.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:16:36.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:16:36.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:16:36.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:36.731323 11919 consensus_peers.cc:597] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 -> Peer 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67:44983): Couldn't send request to peer 2ea1996689fb4cf18b17a35790fba392. Status: Network error: Client connection negotiation failed: client connection to 127.10.40.67:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
08:16:37.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:37.017496 12147 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:16:37.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:37.017804 12147 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:16:37.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:37.017884 12147 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:16:37.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:37.026203 12147 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:16:37.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:37.026377 12147 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.40.67
08:16:37.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.036520 12147 tablet_server_runner.cc:78] Tablet server non-default flags:
08:16:37.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/logs
08:16:37.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:16:37.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:16:37.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:16:37.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.40.84:40955
08:16:37.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:16:37.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:16:37.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/data
08:16:37.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/wal
08:16:37.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:16:37.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:16:37.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.40.67:44983
08:16:37.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:16:37.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:16:37.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:16:37.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/data/info.pb
08:16:37.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.40.67
08:16:37.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45387
08:16:37.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.40.126:45793,127.10.40.125:46051,127.10.40.124:41773
08:16:37.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:16:37.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:16:37.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:16:37.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:16:37.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:16:37.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.40.67
08:16:37.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:16:37.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:16:37.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:16:37.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 70c51312f1186a8ae9f46647160b7730123f7601
08:16:37.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:16:37.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 12 Aug 2026 07:43:21 UTC on bdcb31816ec0
08:16:37.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 13302
08:16:37.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
08:16:37.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.038177 12147 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:16:37.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.040022 12147 file_cache.cc:504] Constructed file cache file cache with capacity 419430
08:16:37.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:37.050266 12153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:16:37.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:37.050509 12154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:16:37.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:37.051350 12156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:16:37.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.051976 12147 server_base.cc:1061] running on GCE node
08:16:37.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.052525 12147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:16:37.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.053465 12147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:16:37.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.054718 12147 hybrid_clock.cc:648] HybridClock initialized: now 1786522597054672 us; error 33 us; skew 500 ppm
08:16:37.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.055130 12147 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:16:37.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.062224 12147 webserver.cc:533] Webserver started at http://127.10.40.67:45387/ using document root <none> and password file <none>
08:16:37.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.063165 12147 fs_manager.cc:362] Metadata directory not provided
08:16:37.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.063335 12147 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:16:37.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.069309 12147 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.000s
08:16:37.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.072337 12162 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:16:37.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.073772 12147 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
08:16:37.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.074035 12147 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/data,/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/wal
08:16:37.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "2ea1996689fb4cf18b17a35790fba392"
08:16:37.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:16:20 on dist-test-slave-gp6n"
08:16:37.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.075002 12147 fs_report.cc:389] FS layout report
08:16:37.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:16:37.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/wal
08:16:37.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/wal
08:16:37.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/data/data
08:16:37.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:16:37.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:16:37.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:16:37.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:16:37.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:16:37.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:16:37.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:16:37.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:16:37.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:16:37.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:16:37.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:16:37.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.094789 12147 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:16:37.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.095722 12147 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:16:37.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.096097 12147 kserver.cc:163] Server-wide thread pool size limit: 3276
08:16:37.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.097239 12147 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
08:16:37.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.099555 12172 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
08:16:37.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.105692 12147 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
08:16:37.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.105829 12147 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.007s	user 0.000s	sys 0.001s
08:16:37.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.105995 12147 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
08:16:37.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.109375 12147 ts_tablet_manager.cc:616] Registered 1 tablets
08:16:37.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.109558 12147 ts_tablet_manager.cc:595] Time spent register tablets: real 0.004s	user 0.001s	sys 0.000s
08:16:37.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.109887 12172 tablet_bootstrap.cc:492] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392: Bootstrap starting.
08:16:37.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.136451 12172 log.cc:826] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392: Log is configured to *not* fsync() on all Append() calls
08:16:37.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.162976 12147 rpc_server.cc:307] RPC server started. Bound to: 127.10.40.67:44983
08:16:37.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.163033 12312 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.40.67:44983 every 8 connection(s)
08:16:37.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.163262 12172 tablet_bootstrap.cc:492] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=12 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
08:16:37.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.163947 12172 tablet_bootstrap.cc:492] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392: Bootstrap complete.
08:16:37.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.164527 12147 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/data/info.pb
08:16:37.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.165782 12172 ts_tablet_manager.cc:1403] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392: Time spent bootstrapping tablet: real 0.056s	user 0.036s	sys 0.016s
08:16:37.169 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:16:37.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.168854 10401 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0lHOBl/build/asan/bin/kudu as pid 12147
08:16:37.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.195436 12172 raft_consensus.cc:359] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:37.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.196880 12172 raft_consensus.cc:740] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ea1996689fb4cf18b17a35790fba392, State: Initialized, Role: FOLLOWER
08:16:37.201 [DEBUG - kudu-nio-7] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 15 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644, tablet=b328961edc164a1289f259240593e73e, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:16:37.202 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.10.40.66:46247
08:16:37.202 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

08:16:37.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.203238 10401 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-task0lHOBl/build/asan/bin/kudu pid 11898
08:16:37.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.208079 12172 consensus_queue.cc:260] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 4.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:37.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.220294 12314 heartbeater.cc:344] Connected to a master server at 127.10.40.126:45793
08:16:37.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.220752 12314 heartbeater.cc:461] Registering TS with master...
08:16:37.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.221678 12314 heartbeater.cc:507] Master 127.10.40.126:45793 requested a full tablet report, sending...
08:16:37.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.225558 12172 ts_tablet_manager.cc:1434] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392: Time spent starting tablet: real 0.060s	user 0.011s	sys 0.003s
08:16:37.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.226686 10451 ts_manager.cc:194] Re-registered known tserver with Master: 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67:44983)
08:16:37.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.228643 12313 heartbeater.cc:344] Connected to a master server at 127.10.40.124:41773
08:16:37.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.228806 12313 heartbeater.cc:461] Registering TS with master...
08:16:37.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.229411 12313 heartbeater.cc:507] Master 127.10.40.124:41773 requested a full tablet report, sending...
08:16:37.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.231343 10648 ts_manager.cc:194] Re-registered known tserver with Master: 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67:44983)
08:16:37.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.232702 10648 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.10.40.67:51727
08:16:37.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.233129 12256 raft_consensus.cc:3060] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 4 FOLLOWER]: Advancing to term 5
08:16:37.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.235879 12313 heartbeater.cc:499] Master 127.10.40.124:41773 was elected leader, sending a full tablet report...
08:16:37.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.244123 12316 heartbeater.cc:344] Connected to a master server at 127.10.40.125:46051
08:16:37.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.244293 12316 heartbeater.cc:461] Registering TS with master...
08:16:37.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.244642 12316 heartbeater.cc:507] Master 127.10.40.125:46051 requested a full tablet report, sending...
08:16:37.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.249223 12057 generic_service.cc:196] Checking for leaks (request via RPC)
08:16:37.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.252283 10550 ts_manager.cc:194] Re-registered known tserver with Master: 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67:44983)
08:16:37.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:37.359347 10401 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0lHOBl/build/asan/bin/kudu with pid 11898
08:16:37.386 [DEBUG - kudu-nio-7] (Connection.java:239) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] handling channelInactive
08:16:37.387 [DEBUG - kudu-nio-7] (Connection.java:709) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] cleaning up while in state READY due to: connection closed
08:16:37.388 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:16:37.388 [DEBUG - Time-limited test] (Connection.java:748) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] connecting to peer
08:16:37.389 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:37.390 [DEBUG - kudu-nio-0] (Connection.java:709) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] cleaning up while in state CONNECTING due to: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:37.390 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:37.406 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:37.406 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983),f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)[L],f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)]}]
08:16:37.406 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=2, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=18), Traces: [0ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [2ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [2ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [17ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@751164409(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@5ea552c1 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2a639a6e -> wakeup thread Time-limited test)) after lookup
08:16:37.407 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] connecting to peer
08:16:37.408 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:37.408 [DEBUG - kudu-nio-1] (Connection.java:709) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] cleaning up while in state CONNECTING due to: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:37.408 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:37.426 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:37.427 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983),f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)[L],f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)]}]
08:16:37.427 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=4, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=38), Traces: [0ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [2ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [2ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [17ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [19ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [20ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [21ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [35ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [38ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@751164409(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@5ea552c1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1192498916 after retry RPC@1897022594), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2a639a6e -> wakeup thread Time-limited test -> (continuation of Deferred@1192498916 after retry RPC@1897022594))) after lookup
08:16:37.427 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] connecting to peer
08:16:37.428 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:37.429 [DEBUG - kudu-nio-2] (Connection.java:709) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] cleaning up while in state CONNECTING due to: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:37.429 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:37.466 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:37.467 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983),f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)[L],f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)]}]
08:16:37.467 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=6, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=78), Traces: [0ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [2ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [2ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [17ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [19ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [20ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [21ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [35ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [38ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [39ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [41ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [41ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [75ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [78ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@751164409(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@5ea552c1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1192498916 after retry RPC@1897022594) -> (continuation of Deferred@1416772384 after retry RPC@925324152), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2a639a6e -> wakeup thread Time-limited test -> (continuation of Deferred@1192498916 after retry RPC@1897022594) -> (continuation of Deferred@1416772384 after retry RPC@925324152))) after lookup
08:16:37.468 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] connecting to peer
08:16:37.469 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:37.469 [DEBUG - kudu-nio-3] (Connection.java:709) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] cleaning up while in state CONNECTING due to: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:37.469 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:37.526 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:37.527 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983),f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)[L],f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)]}]
08:16:37.527 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=8, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=138), Traces: [0ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [2ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [2ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [17ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [19ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [20ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [21ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [35ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [38ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [39ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [41ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [41ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [75ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [78ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [80ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [81ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [81ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [135ms] refreshing cache from master, [135ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [138ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@751164409(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@5ea552c1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1192498916 after retry RPC@1897022594) -> (continuation of Deferred@1416772384 after retry RPC@925324152) -> (continuation of Deferred@1074123412 after retry RPC@1303700863), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2a639a6e -> wakeup thread Time-limited test -> (continuation of Deferred@1192498916 after retry RPC@1897022594) -> (continuation of Deferred@1416772384 after retry RPC@925324152) -> (continuation of Deferred@1074123412 after retry RPC@1303700863))) after lookup
08:16:37.528 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] connecting to peer
08:16:37.529 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:37.529 [DEBUG - kudu-nio-4] (Connection.java:709) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] cleaning up while in state CONNECTING due to: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:37.529 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:38.047 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:38.047 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983),f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)[L],f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)]}]
08:16:38.048 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=10, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=659), Traces: [0ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [2ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [2ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [17ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [19ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [20ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [21ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [35ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [38ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [39ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [41ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [41ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [75ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [78ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [80ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [81ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [81ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [135ms] refreshing cache from master, [135ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [138ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [139ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [141ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [141ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [655ms] refreshing cache from master, [656ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [659ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@751164409(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@5ea552c1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1192498916 after retry RPC@1897022594) -> (continuation of Deferred@1416772384 after retry RPC@925324152) -> (continuation of Deferred@1074123412 after retry RPC@1303700863) -> (continuation of Deferred@46769446 after retry RPC@1748776733), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2a639a6e -> wakeup thread Time-limited test -> (continuation of Deferred@1192498916 after retry RPC@1897022594) -> (continuation of Deferred@1416772384 after retry RPC@925324152) -> (continuation of Deferred@1074123412 after retry RPC@1303700863) -> (continuation of Deferred@46769446 after retry RPC@1748776733))) after lookup
08:16:38.049 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] connecting to peer
08:16:38.050 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:38.050 [DEBUG - kudu-nio-5] (Connection.java:709) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] cleaning up while in state CONNECTING due to: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:38.050 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:38.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:38.740243 12340 raft_consensus.cc:493] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f0e29bd27cb448589ac37e569ad1a164)
08:16:38.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:38.740499 12340 raft_consensus.cc:515] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:38.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:38.740998 12341 raft_consensus.cc:493] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f0e29bd27cb448589ac37e569ad1a164)
08:16:38.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:38.741256 12341 raft_consensus.cc:515] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:38.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:38.741390 12340 leader_election.cc:290] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247), 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67:44983)
08:16:38.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:38.742517 11672 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.40.66:46247: connect: Connection refused (error 111) [suppressed 1 similar messages]
08:16:38.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:38.743435 12341 leader_election.cc:290] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247), f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65:39249)
08:16:38.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:38.744306 11672 leader_election.cc:336] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247): Network error: Client connection negotiation failed: client connection to 127.10.40.66:46247: connect: Connection refused (error 111)
08:16:38.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:38.744843 12163 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.40.66:46247: connect: Connection refused (error 111)
08:16:38.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:38.747691 12163 leader_election.cc:336] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247): Network error: Client connection negotiation failed: client connection to 127.10.40.66:46247: connect: Connection refused (error 111)
08:16:38.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:38.752434 12256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b328961edc164a1289f259240593e73e" candidate_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "2ea1996689fb4cf18b17a35790fba392" is_pre_election: true
08:16:38.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:38.752847 12256 raft_consensus.cc:2468] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2a3f45a9e3b4f2dbda7e2613df89453 in term 5.
08:16:38.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:38.752820 11756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b328961edc164a1289f259240593e73e" candidate_uuid: "2ea1996689fb4cf18b17a35790fba392" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" is_pre_election: true
08:16:38.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:38.753064 11756 raft_consensus.cc:2468] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ea1996689fb4cf18b17a35790fba392 in term 5.
08:16:38.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:38.753444 11672 leader_election.cc:304] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ea1996689fb4cf18b17a35790fba392, f2a3f45a9e3b4f2dbda7e2613df89453; no voters: f0e29bd27cb448589ac37e569ad1a164
08:16:38.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:38.753643 12163 leader_election.cc:304] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ea1996689fb4cf18b17a35790fba392, f2a3f45a9e3b4f2dbda7e2613df89453; no voters: f0e29bd27cb448589ac37e569ad1a164
08:16:38.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:38.753875 12340 raft_consensus.cc:2804] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 5 FOLLOWER]: Leader pre-election won for term 6
08:16:38.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:38.754022 12340 raft_consensus.cc:493] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 5 FOLLOWER]: Starting leader election (detected failure of leader f0e29bd27cb448589ac37e569ad1a164)
08:16:38.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:38.754025 12341 raft_consensus.cc:2804] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 5 FOLLOWER]: Leader pre-election won for term 6
08:16:38.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:38.754096 12340 raft_consensus.cc:3060] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 5 FOLLOWER]: Advancing to term 6
08:16:38.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:38.754145 12341 raft_consensus.cc:493] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 5 FOLLOWER]: Starting leader election (detected failure of leader f0e29bd27cb448589ac37e569ad1a164)
08:16:38.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:38.754243 12341 raft_consensus.cc:3060] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 5 FOLLOWER]: Advancing to term 6
08:16:38.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:38.756567 12340 raft_consensus.cc:515] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:38.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:38.756835 12341 raft_consensus.cc:515] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:38.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:38.757306 12340 leader_election.cc:290] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [CANDIDATE]: Term 6 election: Requested vote from peers f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247), 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67:44983)
08:16:38.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:38.757534 12341 leader_election.cc:290] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [CANDIDATE]: Term 6 election: Requested vote from peers f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247), f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65:39249)
08:16:38.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:38.758287 12256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b328961edc164a1289f259240593e73e" candidate_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "2ea1996689fb4cf18b17a35790fba392"
08:16:38.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:38.758311 11756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b328961edc164a1289f259240593e73e" candidate_uuid: "2ea1996689fb4cf18b17a35790fba392" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453"
08:16:38.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:38.758615 11756 raft_consensus.cc:2393] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 2ea1996689fb4cf18b17a35790fba392 in current term 6: Already voted for candidate f2a3f45a9e3b4f2dbda7e2613df89453 in this term.
08:16:38.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:38.758683 12256 raft_consensus.cc:2393] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f2a3f45a9e3b4f2dbda7e2613df89453 in current term 6: Already voted for candidate 2ea1996689fb4cf18b17a35790fba392 in this term.
08:16:38.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:38.759373 12163 leader_election.cc:336] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247): Network error: Client connection negotiation failed: client connection to 127.10.40.66:46247: connect: Connection refused (error 111)
08:16:38.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:38.759541 11672 leader_election.cc:336] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247): Network error: Client connection negotiation failed: client connection to 127.10.40.66:46247: connect: Connection refused (error 111)
08:16:38.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:38.759629 11672 leader_election.cc:304] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2a3f45a9e3b4f2dbda7e2613df89453; no voters: 2ea1996689fb4cf18b17a35790fba392, f0e29bd27cb448589ac37e569ad1a164
08:16:38.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:38.759742 12163 leader_election.cc:304] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ea1996689fb4cf18b17a35790fba392; no voters: f0e29bd27cb448589ac37e569ad1a164, f2a3f45a9e3b4f2dbda7e2613df89453
08:16:38.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:38.759889 12340 raft_consensus.cc:2749] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
08:16:38.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:38.760119 12341 raft_consensus.cc:2749] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
08:16:40.107 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:40.107 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983),f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)[L],f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)]}]
08:16:40.108 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=12, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=2719), Traces: [0ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [2ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [2ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [17ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [19ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [20ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [21ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [35ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [38ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [39ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [41ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [41ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [75ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [78ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [80ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [81ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [81ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [135ms] refreshing cache from master, [135ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [138ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [139ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [141ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [141ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [655ms] refreshing cache from master, [656ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [659ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [660ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [662ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [662ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [2715ms] refreshing cache from master, [2716ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [2719ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@751164409(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@5ea552c1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1192498916 after retry RPC@1897022594) -> (continuation of Deferred@1416772384 after retry RPC@925324152) -> (continuation of Deferred@1074123412 after retry RPC@1303700863) -> (continuation of Deferred@46769446 after retry RPC@1748776733) -> (continuation of Deferred@1362112339 after retry RPC@1488813751), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2a639a6e -> wakeup thread Time-limited test -> (continuation of Deferred@1192498916 after retry RPC@1897022594) -> (continuation of Deferred@1416772384 after retry RPC@925324152) -> (continuation of Deferred@1074123412 after retry RPC@1303700863) -> (continuation of Deferred@46769446 after retry RPC@1748776733) -> (continuation of Deferred@1362112339 after retry RPC@1488813751))) after lookup
08:16:40.108 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] connecting to peer
08:16:40.109 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:40.110 [DEBUG - kudu-nio-6] (Connection.java:709) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] cleaning up while in state CONNECTING due to: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:40.110 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:40.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:40.706382 12350 raft_consensus.cc:493] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
08:16:40.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:40.706610 12350 raft_consensus.cc:515] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:40.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:40.707370 12350 leader_election.cc:290] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247), 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67:44983)
08:16:40.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:40.708639 12256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b328961edc164a1289f259240593e73e" candidate_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" candidate_term: 7 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "2ea1996689fb4cf18b17a35790fba392" is_pre_election: true
08:16:40.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:40.708868 12256 raft_consensus.cc:2468] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2a3f45a9e3b4f2dbda7e2613df89453 in term 6.
08:16:40.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:40.709538 11672 leader_election.cc:304] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ea1996689fb4cf18b17a35790fba392, f2a3f45a9e3b4f2dbda7e2613df89453; no voters: 
08:16:40.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:40.709851 12350 raft_consensus.cc:2804] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 6 FOLLOWER]: Leader pre-election won for term 7
08:16:40.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:40.709964 12350 raft_consensus.cc:493] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
08:16:40.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:40.710035 12350 raft_consensus.cc:3060] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 6 FOLLOWER]: Advancing to term 7
08:16:40.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:40.710541 11672 leader_election.cc:336] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247): Network error: Client connection negotiation failed: client connection to 127.10.40.66:46247: connect: Connection refused (error 111)
08:16:40.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:40.712212 12350 raft_consensus.cc:515] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:40.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:40.712695 12350 leader_election.cc:290] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [CANDIDATE]: Term 7 election: Requested vote from peers f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247), 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67:44983)
08:16:40.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:40.713564 12256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b328961edc164a1289f259240593e73e" candidate_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" candidate_term: 7 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "2ea1996689fb4cf18b17a35790fba392"
08:16:40.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:40.713766 12256 raft_consensus.cc:3060] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 6 FOLLOWER]: Advancing to term 7
08:16:40.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:40.714152 11672 leader_election.cc:336] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247): Network error: Client connection negotiation failed: client connection to 127.10.40.66:46247: connect: Connection refused (error 111)
08:16:40.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:40.716005 12256 raft_consensus.cc:2468] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2a3f45a9e3b4f2dbda7e2613df89453 in term 7.
08:16:40.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:40.716449 11672 leader_election.cc:304] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ea1996689fb4cf18b17a35790fba392, f2a3f45a9e3b4f2dbda7e2613df89453; no voters: f0e29bd27cb448589ac37e569ad1a164
08:16:40.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:40.716785 12350 raft_consensus.cc:2804] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 7 FOLLOWER]: Leader election won for term 7
08:16:40.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:40.717386 12350 raft_consensus.cc:697] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 7 LEADER]: Becoming Leader. State: Replica: f2a3f45a9e3b4f2dbda7e2613df89453, State: Running, Role: LEADER
08:16:40.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:40.717971 12350 consensus_queue.cc:237] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 5.20, Last appended by leader: 20, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:40.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:40.722224 10648 catalog_manager.cc:5719] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 reported cstate change: term changed from 5 to 7, leader changed from f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66) to f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65). New cstate: current_term: 7 leader_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } health_report { overall_health: HEALTHY } } }
08:16:41.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:41.098512 12256 raft_consensus.cc:1275] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 7 FOLLOWER]: Refusing update from remote peer f2a3f45a9e3b4f2dbda7e2613df89453: Log matching property violated. Preceding OpId in replica: term: 5 index: 20. Preceding OpId from leader: term: 7 index: 21. (index mismatch)
08:16:41.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:41.099378 12350 consensus_queue.cc:1048] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
08:16:41.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:41.109927 11672 consensus_peers.cc:597] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 -> Peer f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247): Couldn't send request to peer f0e29bd27cb448589ac37e569ad1a164. Status: Network error: Client connection negotiation failed: client connection to 127.10.40.66:46247: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
08:16:43.107 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:43.107 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983),f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247),f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)[L]]}]
08:16:43.108 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=14, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=5719), Traces: [0ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [2ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [2ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [17ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [19ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [20ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [21ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [35ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [38ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [39ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [41ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [41ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [75ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [78ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [80ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [81ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [81ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [135ms] refreshing cache from master, [135ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [138ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [139ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [141ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [141ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [655ms] refreshing cache from master, [656ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [659ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [660ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [662ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [662ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [2715ms] refreshing cache from master, [2716ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [2719ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [2720ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [2722ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [2722ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [5715ms] refreshing cache from master, [5715ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [5718ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@751164409(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@5ea552c1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1192498916 after retry RPC@1897022594) -> (continuation of Deferred@1416772384 after retry RPC@925324152) -> (continuation of Deferred@1074123412 after retry RPC@1303700863) -> (continuation of Deferred@46769446 after retry RPC@1748776733) -> (continuation of Deferred@1362112339 after retry RPC@1488813751) -> (continuation of Deferred@1185289269 after retry RPC@1969062544), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2a639a6e -> wakeup thread Time-limited test -> (continuation of Deferred@1192498916 after retry RPC@1897022594) -> (continuation of Deferred@1416772384 after retry RPC@925324152) -> (continuation of Deferred@1074123412 after retry RPC@1303700863) -> (continuation of Deferred@46769446 after retry RPC@1748776733) -> (continuation of Deferred@1362112339 after retry RPC@1488813751) -> (continuation of Deferred@1185289269 after retry RPC@1969062544))) after lookup
08:16:43.109 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] connecting to peer
08:16:43.110 [DEBUG - kudu-nio-7] (Connection.java:757) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] Successfully connected to peer
08:16:43.127 [DEBUG - kudu-nio-7] (Negotiator.java:918) Authenticated connection [id: 0x677b74b4, L:/127.0.0.1:41758 - R:/127.10.40.65:39249] using TOKEN/null
08:16:43.158 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.10.40.66:46247
08:16:43.160 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

08:16:43.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.161445 10401 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0lHOBl/build/asan/bin/kudu
08:16:43.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task0lHOBl/build/asan/bin/kudu
08:16:43.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/wal
08:16:43.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/data
08:16:43.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:16:43.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:16:43.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:16:43.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:16:43.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:16:43.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:16:43.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/logs
08:16:43.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/data/info.pb
08:16:43.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:16:43.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:16:43.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:16:43.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:16:43.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:16:43.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:16:43.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:16:43.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:16:43.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:16:43.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:16:43.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.40.66:46247
08:16:43.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.40.66
08:16:43.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.40.126:45793,127.10.40.125:46051,127.10.40.124:41773
08:16:43.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=33605
08:16:43.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.40.66
08:16:43.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.40.84:40955
08:16:43.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:16:43.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:16:43.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:16:43.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:43.427889 11672 consensus_peers.cc:597] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 -> Peer f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247): Couldn't send request to peer f0e29bd27cb448589ac37e569ad1a164. Status: Network error: Client connection negotiation failed: client connection to 127.10.40.66:46247: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
08:16:43.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:43.525146 12395 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:16:43.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:43.525488 12395 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:16:43.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:43.525576 12395 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:16:43.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:43.534642 12395 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:16:43.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:43.534811 12395 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.40.66
08:16:43.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.545317 12395 tablet_server_runner.cc:78] Tablet server non-default flags:
08:16:43.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/logs
08:16:43.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:16:43.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:16:43.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:16:43.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.40.84:40955
08:16:43.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:16:43.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:16:43.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/data
08:16:43.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/wal
08:16:43.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:16:43.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:16:43.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.40.66:46247
08:16:43.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:16:43.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:16:43.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:16:43.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/data/info.pb
08:16:43.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.40.66
08:16:43.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=33605
08:16:43.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.40.126:45793,127.10.40.125:46051,127.10.40.124:41773
08:16:43.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:16:43.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:16:43.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:16:43.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:16:43.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:16:43.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.40.66
08:16:43.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:16:43.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:16:43.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:16:43.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 70c51312f1186a8ae9f46647160b7730123f7601
08:16:43.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:16:43.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 12 Aug 2026 07:43:21 UTC on bdcb31816ec0
08:16:43.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 13302
08:16:43.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
08:16:43.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.547318 12395 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:16:43.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.549131 12395 file_cache.cc:504] Constructed file cache file cache with capacity 419430
08:16:43.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:43.561450 12404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:16:43.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:43.561419 12403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:16:43.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:43.562033 12407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:16:43.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.562819 12395 server_base.cc:1061] running on GCE node
08:16:43.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.563397 12395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:16:43.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.564415 12395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:16:43.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.565636 12395 hybrid_clock.cc:648] HybridClock initialized: now 1786522603565572 us; error 58 us; skew 500 ppm
08:16:43.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.566068 12395 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:16:43.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.568636 12395 webserver.cc:533] Webserver started at http://127.10.40.66:33605/ using document root <none> and password file <none>
08:16:43.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.569389 12395 fs_manager.cc:362] Metadata directory not provided
08:16:43.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.569492 12395 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:16:43.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.575039 12395 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.000s
08:16:43.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.578055 12413 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:16:43.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.579550 12395 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
08:16:43.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.579690 12395 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/data,/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/wal
08:16:43.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "f0e29bd27cb448589ac37e569ad1a164"
08:16:43.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:16:19 on dist-test-slave-gp6n"
08:16:43.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.580400 12395 fs_report.cc:389] FS layout report
08:16:43.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:16:43.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/wal
08:16:43.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/wal
08:16:43.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/data/data
08:16:43.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:16:43.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:16:43.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:16:43.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:16:43.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:16:43.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:16:43.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:16:43.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:16:43.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:16:43.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:16:43.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:16:43.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.607971 12395 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:16:43.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.608958 12395 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:16:43.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.609225 12395 kserver.cc:163] Server-wide thread pool size limit: 3276
08:16:43.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.610276 12395 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
08:16:43.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.612563 12420 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
08:16:43.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.619321 12395 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
08:16:43.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.619414 12395 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.007s	user 0.001s	sys 0.000s
08:16:43.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.619524 12395 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
08:16:43.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.622648 12395 ts_tablet_manager.cc:616] Registered 1 tablets
08:16:43.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.622746 12395 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.003s	sys 0.000s
08:16:43.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.623276 12420 tablet_bootstrap.cc:492] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164: Bootstrap starting.
08:16:43.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.648495 12420 log.cc:826] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164: Log is configured to *not* fsync() on all Append() calls
08:16:43.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.678306 12420 tablet_bootstrap.cc:492] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=20 ignored=0} inserts{seen=15 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
08:16:43.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.679152 12420 tablet_bootstrap.cc:492] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164: Bootstrap complete.
08:16:43.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.680567 12420 ts_tablet_manager.cc:1403] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164: Time spent bootstrapping tablet: real 0.058s	user 0.038s	sys 0.017s
08:16:43.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.687512 12395 rpc_server.cc:307] RPC server started. Bound to: 127.10.40.66:46247
08:16:43.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.687588 12557 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.40.66:46247 every 8 connection(s)
08:16:43.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.687074 12420 raft_consensus.cc:359] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:43.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.688125 12420 raft_consensus.cc:740] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e29bd27cb448589ac37e569ad1a164, State: Initialized, Role: FOLLOWER
08:16:43.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.689124 12420 consensus_queue.cc:260] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 20, Last appended: 5.20, Last appended by leader: 20, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:43.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.689410 12395 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/data/info.pb
08:16:43.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.690169 10401 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0lHOBl/build/asan/bin/kudu as pid 12395
08:16:43.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.691188 12420 ts_tablet_manager.cc:1434] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164: Time spent starting tablet: real 0.011s	user 0.011s	sys 0.001s
08:16:43.691 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:16:43.713 [DEBUG - kudu-nio-7] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 18 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644, tablet=b328961edc164a1289f259240593e73e, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:16:43.713 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.10.40.65:39249
08:16:43.714 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

08:16:43.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.714677 10401 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-task0lHOBl/build/asan/bin/kudu pid 11652
08:16:43.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.735103 12562 heartbeater.cc:344] Connected to a master server at 127.10.40.126:45793
08:16:43.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.735457 12560 heartbeater.cc:344] Connected to a master server at 127.10.40.124:41773
08:16:43.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.735605 12560 heartbeater.cc:461] Registering TS with master...
08:16:43.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.735603 12562 heartbeater.cc:461] Registering TS with master...
08:16:43.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.736873 12562 heartbeater.cc:507] Master 127.10.40.126:45793 requested a full tablet report, sending...
08:16:43.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.736901 12560 heartbeater.cc:507] Master 127.10.40.124:41773 requested a full tablet report, sending...
08:16:43.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.740388 10648 ts_manager.cc:194] Re-registered known tserver with Master: f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247)
08:16:43.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.740911 10451 ts_manager.cc:194] Re-registered known tserver with Master: f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247)
08:16:43.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.741976 10648 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.10.40.66:50215
08:16:43.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.745023 12560 heartbeater.cc:499] Master 127.10.40.124:41773 was elected leader, sending a full tablet report...
08:16:43.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.748847 12571 heartbeater.cc:344] Connected to a master server at 127.10.40.125:46051
08:16:43.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.749022 12571 heartbeater.cc:461] Registering TS with master...
08:16:43.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.749359 12571 heartbeater.cc:507] Master 127.10.40.125:46051 requested a full tablet report, sending...
08:16:43.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.750859 10550 ts_manager.cc:194] Re-registered known tserver with Master: f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247)
08:16:43.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.752759 11803 generic_service.cc:196] Checking for leaks (request via RPC)
08:16:43.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:43.858260 10401 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0lHOBl/build/asan/bin/kudu with pid 11652
08:16:43.886 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:16:43.886 [DEBUG - kudu-nio-7] (Connection.java:239) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] handling channelInactive
08:16:43.886 [DEBUG - kudu-nio-7] (Connection.java:709) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] cleaning up while in state READY due to: connection closed
08:16:43.887 [DEBUG - Time-limited test] (Connection.java:748) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] connecting to peer
08:16:43.888 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:43.888 [DEBUG - kudu-nio-0] (Connection.java:709) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] cleaning up while in state CONNECTING due to: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:43.888 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:43.907 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:43.907 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983),f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247),f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)[L]]}]
08:16:43.907 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=2, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=21), Traces: [0ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [2ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [2ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [20ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@607185831(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@36b48950 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5fe77677 -> wakeup thread Time-limited test)) after lookup
08:16:43.908 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] connecting to peer
08:16:43.909 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:43.909 [DEBUG - kudu-nio-1] (Connection.java:709) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] cleaning up while in state CONNECTING due to: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:43.909 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:43.927 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:43.928 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983),f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247),f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)[L]]}]
08:16:43.928 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=4, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=42), Traces: [0ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [2ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [2ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [20ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [21ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [23ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [23ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [37ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [41ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@607185831(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@36b48950 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@485935037 after retry RPC@2017873562), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5fe77677 -> wakeup thread Time-limited test -> (continuation of Deferred@485935037 after retry RPC@2017873562))) after lookup
08:16:43.929 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] connecting to peer
08:16:43.931 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:43.931 [DEBUG - kudu-nio-2] (Connection.java:709) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] cleaning up while in state CONNECTING due to: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:43.931 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:43.947 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:43.948 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983),f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247),f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)[L]]}]
08:16:43.948 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=6, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=61), Traces: [0ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [2ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [2ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [20ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [21ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [23ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [23ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [37ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [41ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [43ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [45ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [45ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [57ms] refreshing cache from master, [57ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [61ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@607185831(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@36b48950 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@485935037 after retry RPC@2017873562) -> (continuation of Deferred@81913196 after retry RPC@1318668788), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5fe77677 -> wakeup thread Time-limited test -> (continuation of Deferred@485935037 after retry RPC@2017873562) -> (continuation of Deferred@81913196 after retry RPC@1318668788))) after lookup
08:16:43.949 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] connecting to peer
08:16:43.950 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:43.950 [DEBUG - kudu-nio-3] (Connection.java:709) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] cleaning up while in state CONNECTING due to: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:43.950 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:44.007 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:44.008 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983),f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247),f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)[L]]}]
08:16:44.008 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=8, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=122), Traces: [0ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [2ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [2ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [20ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [21ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [23ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [23ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [37ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [41ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [43ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [45ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [45ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [57ms] refreshing cache from master, [57ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [61ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [62ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [64ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [64ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [117ms] refreshing cache from master, [117ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [121ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@607185831(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@36b48950 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@485935037 after retry RPC@2017873562) -> (continuation of Deferred@81913196 after retry RPC@1318668788) -> (continuation of Deferred@2047597220 after retry RPC@2119822072), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5fe77677 -> wakeup thread Time-limited test -> (continuation of Deferred@485935037 after retry RPC@2017873562) -> (continuation of Deferred@81913196 after retry RPC@1318668788) -> (continuation of Deferred@2047597220 after retry RPC@2119822072))) after lookup
08:16:44.009 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] connecting to peer
08:16:44.011 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:44.011 [DEBUG - kudu-nio-4] (Connection.java:709) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] cleaning up while in state CONNECTING due to: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:44.011 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:44.327 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:44.328 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983),f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247),f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)[L]]}]
08:16:44.328 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=10, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=442), Traces: [0ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [2ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [2ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [20ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [21ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [23ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [23ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [37ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [41ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [43ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [45ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [45ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [57ms] refreshing cache from master, [57ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [61ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [62ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [64ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [64ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [117ms] refreshing cache from master, [117ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [121ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [123ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [125ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [125ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [437ms] refreshing cache from master, [437ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [441ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@607185831(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@36b48950 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@485935037 after retry RPC@2017873562) -> (continuation of Deferred@81913196 after retry RPC@1318668788) -> (continuation of Deferred@2047597220 after retry RPC@2119822072) -> (continuation of Deferred@168650167 after retry RPC@993153060), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5fe77677 -> wakeup thread Time-limited test -> (continuation of Deferred@485935037 after retry RPC@2017873562) -> (continuation of Deferred@81913196 after retry RPC@1318668788) -> (continuation of Deferred@2047597220 after retry RPC@2119822072) -> (continuation of Deferred@168650167 after retry RPC@993153060))) after lookup
08:16:44.329 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] connecting to peer
08:16:44.330 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:44.330 [DEBUG - kudu-nio-5] (Connection.java:709) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] cleaning up while in state CONNECTING due to: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:44.331 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:44.987 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:44.987 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983),f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247),f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)[L]]}]
08:16:44.988 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=12, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=1101), Traces: [0ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [2ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [2ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [20ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [21ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [23ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [23ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [37ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [41ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [43ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [45ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [45ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [57ms] refreshing cache from master, [57ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [61ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [62ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [64ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [64ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [117ms] refreshing cache from master, [117ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [121ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [123ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [125ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [125ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [437ms] refreshing cache from master, [437ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [441ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [443ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [445ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [445ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [1097ms] refreshing cache from master, [1097ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [1100ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@607185831(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@36b48950 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@485935037 after retry RPC@2017873562) -> (continuation of Deferred@81913196 after retry RPC@1318668788) -> (continuation of Deferred@2047597220 after retry RPC@2119822072) -> (continuation of Deferred@168650167 after retry RPC@993153060) -> (continuation of Deferred@1587205652 after retry RPC@1996492435), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5fe77677 -> wakeup thread Time-limited test -> (continuation of Deferred@485935037 after retry RPC@2017873562) -> (continuation of Deferred@81913196 after retry RPC@1318668788) -> (continuation of Deferred@2047597220 after retry RPC@2119822072) -> (continuation of Deferred@168650167 after retry RPC@993153060) -> (continuation of Deferred@1587205652 after retry RPC@1996492435))) after lookup
08:16:44.988 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] connecting to peer
08:16:44.989 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:44.989 [DEBUG - kudu-nio-6] (Connection.java:709) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] cleaning up while in state CONNECTING due to: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:44.990 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:45.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:45.141157 12580 raft_consensus.cc:493] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f2a3f45a9e3b4f2dbda7e2613df89453)
08:16:45.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:45.141485 12580 raft_consensus.cc:515] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:45.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:45.142277 12580 leader_election.cc:290] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247), f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65:39249)
08:16:45.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:45.144906 12163 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.40.65:39249: connect: Connection refused (error 111) [suppressed 1 similar messages]
08:16:45.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:45.146337 12163 leader_election.cc:336] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65:39249): Network error: Client connection negotiation failed: client connection to 127.10.40.65:39249: connect: Connection refused (error 111)
08:16:45.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:45.151301 12499 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b328961edc164a1289f259240593e73e" candidate_uuid: "2ea1996689fb4cf18b17a35790fba392" candidate_term: 8 candidate_status { last_received { term: 7 index: 24 } } ignore_live_leader: false dest_uuid: "f0e29bd27cb448589ac37e569ad1a164" is_pre_election: true
08:16:45.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:45.151779 12499 raft_consensus.cc:2468] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ea1996689fb4cf18b17a35790fba392 in term 5.
08:16:45.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:45.152704 12163 leader_election.cc:304] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ea1996689fb4cf18b17a35790fba392, f0e29bd27cb448589ac37e569ad1a164; no voters: f2a3f45a9e3b4f2dbda7e2613df89453
08:16:45.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:45.153034 12580 raft_consensus.cc:2804] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 7 FOLLOWER]: Leader pre-election won for term 8
08:16:45.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:45.153163 12580 raft_consensus.cc:493] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 7 FOLLOWER]: Starting leader election (detected failure of leader f2a3f45a9e3b4f2dbda7e2613df89453)
08:16:45.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:45.153239 12580 raft_consensus.cc:3060] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 7 FOLLOWER]: Advancing to term 8
08:16:45.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:45.155448 12580 raft_consensus.cc:515] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:45.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:45.156032 12580 leader_election.cc:290] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [CANDIDATE]: Term 8 election: Requested vote from peers f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247), f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65:39249)
08:16:45.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:45.156556 12499 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b328961edc164a1289f259240593e73e" candidate_uuid: "2ea1996689fb4cf18b17a35790fba392" candidate_term: 8 candidate_status { last_received { term: 7 index: 24 } } ignore_live_leader: false dest_uuid: "f0e29bd27cb448589ac37e569ad1a164"
08:16:45.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:45.156785 12499 raft_consensus.cc:3060] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 5 FOLLOWER]: Advancing to term 8
08:16:45.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:45.157605 12163 leader_election.cc:336] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65:39249): Network error: Client connection negotiation failed: client connection to 127.10.40.65:39249: connect: Connection refused (error 111)
08:16:45.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:45.161216 12499 raft_consensus.cc:2468] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ea1996689fb4cf18b17a35790fba392 in term 8.
08:16:45.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:45.161737 12163 leader_election.cc:304] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ea1996689fb4cf18b17a35790fba392, f0e29bd27cb448589ac37e569ad1a164; no voters: f2a3f45a9e3b4f2dbda7e2613df89453
08:16:45.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:45.162040 12580 raft_consensus.cc:2804] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 8 FOLLOWER]: Leader election won for term 8
08:16:45.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:45.162583 12580 raft_consensus.cc:697] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 8 LEADER]: Becoming Leader. State: Replica: 2ea1996689fb4cf18b17a35790fba392, State: Running, Role: LEADER
08:16:45.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:45.163197 12580 consensus_queue.cc:237] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 7.24, Last appended by leader: 24, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:45.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:45.167608 10648 catalog_manager.cc:5719] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 reported cstate change: term changed from 7 to 8, leader changed from f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65) to 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67). New cstate: current_term: 8 leader_uuid: "2ea1996689fb4cf18b17a35790fba392" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } health_report { overall_health: UNKNOWN } } }
08:16:45.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:45.629913 12499 raft_consensus.cc:1275] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 8 FOLLOWER]: Refusing update from remote peer 2ea1996689fb4cf18b17a35790fba392: Log matching property violated. Preceding OpId in replica: term: 5 index: 20. Preceding OpId from leader: term: 8 index: 25. (index mismatch)
08:16:45.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:45.631076 12580 consensus_queue.cc:1048] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 20, Time since last communication: 0.000s
08:16:45.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:45.644074 12163 consensus_peers.cc:597] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 -> Peer f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65:39249): Couldn't send request to peer f2a3f45a9e3b4f2dbda7e2613df89453. Status: Network error: Client connection negotiation failed: client connection to 127.10.40.65:39249: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
08:16:48.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:48.187424 12163 consensus_peers.cc:597] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 -> Peer f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65:39249): Couldn't send request to peer f2a3f45a9e3b4f2dbda7e2613df89453. Status: Network error: Client connection negotiation failed: client connection to 127.10.40.65:39249: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
08:16:48.707 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:48.707 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)[L],f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247),f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)]}]
08:16:48.708 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=14, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=4821), Traces: [0ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [2ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [2ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [20ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [21ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [23ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [23ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [37ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [41ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [43ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [45ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [45ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [57ms] refreshing cache from master, [57ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [61ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [62ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [64ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [64ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [117ms] refreshing cache from master, [117ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [121ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [123ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [125ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [125ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [437ms] refreshing cache from master, [437ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [441ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [443ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [445ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [445ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [1097ms] refreshing cache from master, [1097ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [1100ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [1102ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [1103ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [1104ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [4817ms] refreshing cache from master, [4818ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [4821ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@607185831(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@36b48950 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@485935037 after retry RPC@2017873562) -> (continuation of Deferred@81913196 after retry RPC@1318668788) -> (continuation of Deferred@2047597220 after retry RPC@2119822072) -> (continuation of Deferred@168650167 after retry RPC@993153060) -> (continuation of Deferred@1587205652 after retry RPC@1996492435) -> (continuation of Deferred@2124189709 after retry RPC@153054683), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5fe77677 -> wakeup thread Time-limited test -> (continuation of Deferred@485935037 after retry RPC@2017873562) -> (continuation of Deferred@81913196 after retry RPC@1318668788) -> (continuation of Deferred@2047597220 after retry RPC@2119822072) -> (continuation of Deferred@168650167 after retry RPC@993153060) -> (continuation of Deferred@1587205652 after retry RPC@1996492435) -> (continuation of Deferred@2124189709 after retry RPC@153054683))) after lookup
08:16:48.709 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] connecting to peer
08:16:48.710 [DEBUG - kudu-nio-7] (Connection.java:757) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] Successfully connected to peer
08:16:48.730 [DEBUG - kudu-nio-7] (Negotiator.java:918) Authenticated connection [id: 0x06c2a4c4, L:/127.0.0.1:53684 - R:/127.10.40.67:44983] using TOKEN/null
08:16:48.756 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.10.40.65:39249
08:16:48.756 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

08:16:48.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:48.757709 10401 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0lHOBl/build/asan/bin/kudu
08:16:48.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task0lHOBl/build/asan/bin/kudu
08:16:48.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/wal
08:16:48.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/data
08:16:48.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:16:48.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:16:48.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:16:48.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:16:48.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:16:48.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:16:48.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/logs
08:16:48.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/data/info.pb
08:16:48.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:16:48.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:16:48.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:16:48.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:16:48.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:16:48.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:16:48.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:16:48.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:16:48.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:16:48.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:16:48.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.40.65:39249
08:16:48.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.40.65
08:16:48.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.40.126:45793,127.10.40.125:46051,127.10.40.124:41773
08:16:48.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=41939
08:16:48.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.40.65
08:16:48.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.40.84:40955
08:16:48.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:16:48.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:16:48.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:16:49.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:49.123554 12631 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:16:49.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:49.123888 12631 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:16:49.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:49.123986 12631 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:16:49.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:49.133039 12631 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:16:49.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:49.133230 12631 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.40.65
08:16:49.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.145200 12631 tablet_server_runner.cc:78] Tablet server non-default flags:
08:16:49.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/logs
08:16:49.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:16:49.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:16:49.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:16:49.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.40.84:40955
08:16:49.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:16:49.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:16:49.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/data
08:16:49.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/wal
08:16:49.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:16:49.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:16:49.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.40.65:39249
08:16:49.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:16:49.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:16:49.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:16:49.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/data/info.pb
08:16:49.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.40.65
08:16:49.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=41939
08:16:49.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.40.126:45793,127.10.40.125:46051,127.10.40.124:41773
08:16:49.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:16:49.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:16:49.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:16:49.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:16:49.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:16:49.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.40.65
08:16:49.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:16:49.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:16:49.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:16:49.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 70c51312f1186a8ae9f46647160b7730123f7601
08:16:49.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:16:49.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 12 Aug 2026 07:43:21 UTC on bdcb31816ec0
08:16:49.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 13302
08:16:49.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
08:16:49.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.147135 12631 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:16:49.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.149168 12631 file_cache.cc:504] Constructed file cache file cache with capacity 419430
08:16:49.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:49.160511 12638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:16:49.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:49.160619 12636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:16:49.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:49.160619 12640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:16:49.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.163003 12631 server_base.cc:1061] running on GCE node
08:16:49.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.163623 12631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:16:49.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.164741 12631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:16:49.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.166013 12631 hybrid_clock.cc:648] HybridClock initialized: now 1786522609165942 us; error 60 us; skew 500 ppm
08:16:49.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.166333 12631 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:16:49.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.169099 12631 webserver.cc:533] Webserver started at http://127.10.40.65:41939/ using document root <none> and password file <none>
08:16:49.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.170029 12631 fs_manager.cc:362] Metadata directory not provided
08:16:49.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.170214 12631 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:16:49.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.176602 12631 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.001s
08:16:49.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.179905 12647 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:16:49.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.181438 12631 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.004s
08:16:49.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.181573 12631 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/data,/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/wal
08:16:49.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "f2a3f45a9e3b4f2dbda7e2613df89453"
08:16:49.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:16:19 on dist-test-slave-gp6n"
08:16:49.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.182273 12631 fs_report.cc:389] FS layout report
08:16:49.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:16:49.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/wal
08:16:49.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/wal
08:16:49.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/data/data
08:16:49.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:16:49.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:16:49.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:16:49.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:16:49.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:16:49.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:16:49.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:16:49.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:16:49.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:16:49.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:16:49.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:16:49.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.245369 12631 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:16:49.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.246285 12631 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:16:49.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.246652 12631 kserver.cc:163] Server-wide thread pool size limit: 3276
08:16:49.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.247679 12631 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
08:16:49.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.250147 12654 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
08:16:49.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.257863 12631 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
08:16:49.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.257961 12631 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.008s	user 0.001s	sys 0.000s
08:16:49.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.258075 12631 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
08:16:49.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.261210 12631 ts_tablet_manager.cc:616] Registered 1 tablets
08:16:49.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.261373 12631 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.001s	sys 0.000s
08:16:49.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.261767 12654 tablet_bootstrap.cc:492] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453: Bootstrap starting.
08:16:49.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.290180 12654 log.cc:826] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453: Log is configured to *not* fsync() on all Append() calls
08:16:49.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.309177 12631 rpc_server.cc:307] RPC server started. Bound to: 127.10.40.65:39249
08:16:49.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.309233 12798 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.40.65:39249 every 8 connection(s)
08:16:49.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.310750 12631 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/data/info.pb
08:16:49.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.316327 10401 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0lHOBl/build/asan/bin/kudu as pid 12631
08:16:49.317 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:16:49.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.334317 12654 tablet_bootstrap.cc:492] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453: Bootstrap replayed 1/1 log segments. Stats: ops{read=24 overwritten=0 applied=24 ignored=0} inserts{seen=18 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
08:16:49.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.335216 12654 tablet_bootstrap.cc:492] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453: Bootstrap complete.
08:16:49.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.337363 12654 ts_tablet_manager.cc:1403] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453: Time spent bootstrapping tablet: real 0.076s	user 0.063s	sys 0.009s
08:16:49.341 [DEBUG - kudu-nio-7] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 21 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644, tablet=b328961edc164a1289f259240593e73e, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:16:49.342 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.10.40.67:44983
08:16:49.343 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

08:16:49.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.343528 10401 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-task0lHOBl/build/asan/bin/kudu pid 12147
08:16:49.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.347092 12654 raft_consensus.cc:359] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:49.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.348435 12654 raft_consensus.cc:740] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2a3f45a9e3b4f2dbda7e2613df89453, State: Initialized, Role: FOLLOWER
08:16:49.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.350360 12654 consensus_queue.cc:260] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 7.24, Last appended by leader: 24, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:49.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.362757 12654 ts_tablet_manager.cc:1434] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453: Time spent starting tablet: real 0.025s	user 0.017s	sys 0.005s
08:16:49.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.366315 12307 generic_service.cc:196] Checking for leaks (request via RPC)
08:16:49.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.370625 12800 heartbeater.cc:344] Connected to a master server at 127.10.40.126:45793
08:16:49.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.371186 12800 heartbeater.cc:461] Registering TS with master...
08:16:49.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.372818 12800 heartbeater.cc:507] Master 127.10.40.126:45793 requested a full tablet report, sending...
08:16:49.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.373013 12799 heartbeater.cc:344] Connected to a master server at 127.10.40.124:41773
08:16:49.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.373163 12799 heartbeater.cc:461] Registering TS with master...
08:16:49.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.373829 12799 heartbeater.cc:507] Master 127.10.40.124:41773 requested a full tablet report, sending...
08:16:49.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.376555 10647 ts_manager.cc:194] Re-registered known tserver with Master: f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65:39249)
08:16:49.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.377477 10451 ts_manager.cc:194] Re-registered known tserver with Master: f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65:39249)
08:16:49.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.377681 12802 heartbeater.cc:344] Connected to a master server at 127.10.40.125:46051
08:16:49.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.377971 12802 heartbeater.cc:461] Registering TS with master...
08:16:49.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.378185 10647 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.10.40.65:56711
08:16:49.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.378602 12802 heartbeater.cc:507] Master 127.10.40.125:46051 requested a full tablet report, sending...
08:16:49.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.380301 10549 ts_manager.cc:194] Re-registered known tserver with Master: f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65:39249)
08:16:49.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.381006 12799 heartbeater.cc:499] Master 127.10.40.124:41773 was elected leader, sending a full tablet report...
08:16:49.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:49.477910 10401 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0lHOBl/build/asan/bin/kudu with pid 12147
08:16:49.507 [DEBUG - kudu-nio-7] (Connection.java:239) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] handling channelInactive
08:16:49.507 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] cleaning up while in state READY due to: connection closed
08:16:49.507 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:16:49.508 [DEBUG - Time-limited test] (Connection.java:748) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] connecting to peer
08:16:49.509 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:49.509 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] cleaning up while in state CONNECTING due to: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:49.509 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:49.526 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:49.527 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)[L],f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247),f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)]}]
08:16:49.527 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=2, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=19), Traces: [0ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [2ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [2ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [16ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [19ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@408179073(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@133da0fe -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@357a3b35 -> wakeup thread Time-limited test)) after lookup
08:16:49.527 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] connecting to peer
08:16:49.528 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:49.528 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] cleaning up while in state CONNECTING due to: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:49.529 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:49.546 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:49.546 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)[L],f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247),f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)]}]
08:16:49.547 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=4, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=39), Traces: [0ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [2ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [2ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [16ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [19ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [20ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [21ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [22ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [39ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@408179073(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@133da0fe -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1831706563 after retry RPC@1905649481), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@357a3b35 -> wakeup thread Time-limited test -> (continuation of Deferred@1831706563 after retry RPC@1905649481))) after lookup
08:16:49.547 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] connecting to peer
08:16:49.548 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:49.548 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] cleaning up while in state CONNECTING due to: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:49.549 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:49.586 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:49.587 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)[L],f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247),f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)]}]
08:16:49.587 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=6, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=79), Traces: [0ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [2ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [2ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [16ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [19ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [20ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [21ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [22ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [39ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [40ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [42ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [42ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [76ms] refreshing cache from master, [76ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [79ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@408179073(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@133da0fe -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1831706563 after retry RPC@1905649481) -> (continuation of Deferred@761473177 after retry RPC@1930571219), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@357a3b35 -> wakeup thread Time-limited test -> (continuation of Deferred@1831706563 after retry RPC@1905649481) -> (continuation of Deferred@761473177 after retry RPC@1930571219))) after lookup
08:16:49.588 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] connecting to peer
08:16:49.589 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:49.589 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] cleaning up while in state CONNECTING due to: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:49.589 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:49.666 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:49.667 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)[L],f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247),f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)]}]
08:16:49.667 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=8, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=159), Traces: [0ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [2ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [2ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [16ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [19ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [20ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [21ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [22ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [39ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [40ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [42ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [42ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [76ms] refreshing cache from master, [76ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [79ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [81ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [82ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [82ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [156ms] refreshing cache from master, [156ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [159ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@408179073(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@133da0fe -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1831706563 after retry RPC@1905649481) -> (continuation of Deferred@761473177 after retry RPC@1930571219) -> (continuation of Deferred@1937625974 after retry RPC@291705983), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@357a3b35 -> wakeup thread Time-limited test -> (continuation of Deferred@1831706563 after retry RPC@1905649481) -> (continuation of Deferred@761473177 after retry RPC@1930571219) -> (continuation of Deferred@1937625974 after retry RPC@291705983))) after lookup
08:16:49.668 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] connecting to peer
08:16:49.669 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:49.669 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] cleaning up while in state CONNECTING due to: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:49.669 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:49.846 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:49.846 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)[L],f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247),f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)]}]
08:16:49.847 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=10, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=339), Traces: [0ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [2ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [2ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [16ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [19ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [20ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [21ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [22ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [39ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [40ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [42ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [42ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [76ms] refreshing cache from master, [76ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [79ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [81ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [82ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [82ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [156ms] refreshing cache from master, [156ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [159ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [161ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [162ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [162ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [336ms] refreshing cache from master, [336ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [339ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@408179073(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@133da0fe -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1831706563 after retry RPC@1905649481) -> (continuation of Deferred@761473177 after retry RPC@1930571219) -> (continuation of Deferred@1937625974 after retry RPC@291705983) -> (continuation of Deferred@1236458603 after retry RPC@1081862606), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@357a3b35 -> wakeup thread Time-limited test -> (continuation of Deferred@1831706563 after retry RPC@1905649481) -> (continuation of Deferred@761473177 after retry RPC@1930571219) -> (continuation of Deferred@1937625974 after retry RPC@291705983) -> (continuation of Deferred@1236458603 after retry RPC@1081862606))) after lookup
08:16:49.847 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] connecting to peer
08:16:49.848 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:49.849 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] cleaning up while in state CONNECTING due to: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:49.849 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983
08:16:50.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:50.256687 12816 raft_consensus.cc:493] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 2ea1996689fb4cf18b17a35790fba392)
08:16:50.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:50.256896 12816 raft_consensus.cc:515] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:50.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:50.258504 12816 leader_election.cc:290] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67:44983), f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65:39249)
08:16:50.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:50.259814 12414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.40.67:44983: connect: Connection refused (error 111)
08:16:50.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:50.263048 12414 leader_election.cc:336] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67:44983): Network error: Client connection negotiation failed: client connection to 127.10.40.67:44983: connect: Connection refused (error 111)
08:16:50.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:50.267663 12740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b328961edc164a1289f259240593e73e" candidate_uuid: "f0e29bd27cb448589ac37e569ad1a164" candidate_term: 9 candidate_status { last_received { term: 8 index: 28 } } ignore_live_leader: false dest_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" is_pre_election: true
08:16:50.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:50.268229 12740 raft_consensus.cc:2468] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e29bd27cb448589ac37e569ad1a164 in term 7.
08:16:50.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:50.269268 12414 leader_election.cc:304] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f0e29bd27cb448589ac37e569ad1a164, f2a3f45a9e3b4f2dbda7e2613df89453; no voters: 2ea1996689fb4cf18b17a35790fba392
08:16:50.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:50.269688 12816 raft_consensus.cc:2804] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 8 FOLLOWER]: Leader pre-election won for term 9
08:16:50.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:50.269855 12816 raft_consensus.cc:493] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 2ea1996689fb4cf18b17a35790fba392)
08:16:50.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:50.270000 12816 raft_consensus.cc:3060] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 8 FOLLOWER]: Advancing to term 9
08:16:50.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:50.272627 12816 raft_consensus.cc:515] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:50.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:50.273324 12816 leader_election.cc:290] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [CANDIDATE]: Term 9 election: Requested vote from peers 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67:44983), f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65:39249)
08:16:50.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:50.274187 12740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b328961edc164a1289f259240593e73e" candidate_uuid: "f0e29bd27cb448589ac37e569ad1a164" candidate_term: 9 candidate_status { last_received { term: 8 index: 28 } } ignore_live_leader: false dest_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453"
08:16:50.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:50.274444 12740 raft_consensus.cc:3060] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 7 FOLLOWER]: Advancing to term 9
08:16:50.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:50.274931 12414 leader_election.cc:336] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67:44983): Network error: Client connection negotiation failed: client connection to 127.10.40.67:44983: connect: Connection refused (error 111)
08:16:50.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:50.278448 12740 raft_consensus.cc:2468] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e29bd27cb448589ac37e569ad1a164 in term 9.
08:16:50.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:50.278945 12414 leader_election.cc:304] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f0e29bd27cb448589ac37e569ad1a164, f2a3f45a9e3b4f2dbda7e2613df89453; no voters: 2ea1996689fb4cf18b17a35790fba392
08:16:50.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:50.279273 12816 raft_consensus.cc:2804] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 9 FOLLOWER]: Leader election won for term 9
08:16:50.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:50.279765 12816 raft_consensus.cc:697] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 9 LEADER]: Becoming Leader. State: Replica: f0e29bd27cb448589ac37e569ad1a164, State: Running, Role: LEADER
08:16:50.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:50.280443 12816 consensus_queue.cc:237] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 8.28, Last appended by leader: 28, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:50.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:50.284719 10647 catalog_manager.cc:5719] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 reported cstate change: term changed from 8 to 9, leader changed from 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67) to f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66). New cstate: current_term: 9 leader_uuid: "f0e29bd27cb448589ac37e569ad1a164" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } health_report { overall_health: UNKNOWN } } }
08:16:50.567 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:50.567 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983),f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)[L],f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)]}]
08:16:50.567 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=12, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=1059), Traces: [0ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [2ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [2ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [16ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [19ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [20ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [21ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [22ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [39ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [40ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [42ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [42ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [76ms] refreshing cache from master, [76ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [79ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [81ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [82ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [82ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [156ms] refreshing cache from master, [156ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [159ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [161ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [162ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [162ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [336ms] refreshing cache from master, [336ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [339ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [340ms] sending RPC to server 2ea1996689fb4cf18b17a35790fba392, [342ms] received response from server 2ea1996689fb4cf18b17a35790fba392: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [342ms] delaying RPC due to: Network error: Failed to connect to peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983): Connection refused: /127.10.40.67:44983, [1056ms] refreshing cache from master, [1056ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [1059ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@408179073(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@133da0fe -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1831706563 after retry RPC@1905649481) -> (continuation of Deferred@761473177 after retry RPC@1930571219) -> (continuation of Deferred@1937625974 after retry RPC@291705983) -> (continuation of Deferred@1236458603 after retry RPC@1081862606) -> (continuation of Deferred@1547984110 after retry RPC@1131346737), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@357a3b35 -> wakeup thread Time-limited test -> (continuation of Deferred@1831706563 after retry RPC@1905649481) -> (continuation of Deferred@761473177 after retry RPC@1930571219) -> (continuation of Deferred@1937625974 after retry RPC@291705983) -> (continuation of Deferred@1236458603 after retry RPC@1081862606) -> (continuation of Deferred@1547984110 after retry RPC@1131346737))) after lookup
08:16:50.568 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] connecting to peer
08:16:50.569 [DEBUG - kudu-nio-6] (Connection.java:757) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] Successfully connected to peer
08:16:50.599 [DEBUG - kudu-nio-6] (Negotiator.java:918) Authenticated connection [id: 0xe124ee3a, L:/127.0.0.1:56220 - R:/127.10.40.66:46247] using TOKEN/null
08:16:50.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:50.605962 12740 raft_consensus.cc:1275] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 9 FOLLOWER]: Refusing update from remote peer f0e29bd27cb448589ac37e569ad1a164: Log matching property violated. Preceding OpId in replica: term: 7 index: 24. Preceding OpId from leader: term: 9 index: 30. (index mismatch)
08:16:50.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:50.606280 12414 consensus_peers.cc:597] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 -> Peer 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67:44983): Couldn't send request to peer 2ea1996689fb4cf18b17a35790fba392. Status: Network error: Client connection negotiation failed: client connection to 127.10.40.67:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
08:16:50.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:50.607024 12821 consensus_queue.cc:1048] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 24, Time since last communication: 0.000s
08:16:50.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:50.614429 12838 mvcc.cc:204] Tried to move back new op lower bound from 7317596605372788736 to 7317596590752780288. Current Snapshot: MvccSnapshot[applied={T|T < 7317596605372788736}]
08:16:50.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:50.623422 12836 mvcc.cc:204] Tried to move back new op lower bound from 7317596613031256064 to 7317596611712782336. Current Snapshot: MvccSnapshot[applied={T|T < 7317596613031256064}]
08:16:50.647 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.10.40.67:44983
08:16:50.647 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

08:16:50.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:50.648424 10401 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0lHOBl/build/asan/bin/kudu
08:16:50.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task0lHOBl/build/asan/bin/kudu
08:16:50.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/wal
08:16:50.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/data
08:16:50.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:16:50.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:16:50.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:16:50.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:16:50.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:16:50.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:16:50.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/logs
08:16:50.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/data/info.pb
08:16:50.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:16:50.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:16:50.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:16:50.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:16:50.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:16:50.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:16:50.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:16:50.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:16:50.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:16:50.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:16:50.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.40.67:44983
08:16:50.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.40.67
08:16:50.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.40.126:45793,127.10.40.125:46051,127.10.40.124:41773
08:16:50.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45387
08:16:50.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.40.67
08:16:50.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.40.84:40955
08:16:50.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:16:50.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:16:50.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:16:51.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:51.013590 12846 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:16:51.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:51.014037 12846 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:16:51.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:51.014153 12846 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:16:51.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:51.024147 12846 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:16:51.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:51.024312 12846 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.40.67
08:16:51.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.034889 12846 tablet_server_runner.cc:78] Tablet server non-default flags:
08:16:51.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/logs
08:16:51.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:16:51.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:16:51.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:16:51.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.40.84:40955
08:16:51.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:16:51.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:16:51.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/data
08:16:51.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/wal
08:16:51.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:16:51.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:16:51.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.40.67:44983
08:16:51.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:16:51.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:16:51.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:16:51.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/data/info.pb
08:16:51.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.40.67
08:16:51.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45387
08:16:51.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.40.126:45793,127.10.40.125:46051,127.10.40.124:41773
08:16:51.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:16:51.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:16:51.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:16:51.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:16:51.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:16:51.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.40.67
08:16:51.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:16:51.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:16:51.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:16:51.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 70c51312f1186a8ae9f46647160b7730123f7601
08:16:51.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:16:51.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 12 Aug 2026 07:43:21 UTC on bdcb31816ec0
08:16:51.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 13302
08:16:51.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
08:16:51.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.036526 12846 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:16:51.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.038288 12846 file_cache.cc:504] Constructed file cache file cache with capacity 419430
08:16:51.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:51.047981 12854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:16:51.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:51.048027 12855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:16:51.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:51.048949 12857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:16:51.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.049618 12846 server_base.cc:1061] running on GCE node
08:16:51.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.050179 12846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:16:51.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.051270 12846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:16:51.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.052498 12846 hybrid_clock.cc:648] HybridClock initialized: now 1786522611052436 us; error 59 us; skew 500 ppm
08:16:51.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.052775 12846 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:16:51.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.055086 12846 webserver.cc:533] Webserver started at http://127.10.40.67:45387/ using document root <none> and password file <none>
08:16:51.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.055850 12846 fs_manager.cc:362] Metadata directory not provided
08:16:51.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.055955 12846 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:16:51.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.061647 12846 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.000s
08:16:51.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.064505 12868 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:16:51.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.065900 12846 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
08:16:51.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.066047 12846 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/data,/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/wal
08:16:51.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "2ea1996689fb4cf18b17a35790fba392"
08:16:51.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:16:20 on dist-test-slave-gp6n"
08:16:51.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.066996 12846 fs_report.cc:389] FS layout report
08:16:51.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:16:51.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/wal
08:16:51.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/wal
08:16:51.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/data/data
08:16:51.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:16:51.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:16:51.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:16:51.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:16:51.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:16:51.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:16:51.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:16:51.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:16:51.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:16:51.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:16:51.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:16:51.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.098930 12846 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:16:51.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.099840 12846 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:16:51.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.100113 12846 kserver.cc:163] Server-wide thread pool size limit: 3276
08:16:51.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.101287 12846 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
08:16:51.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.103679 12876 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
08:16:51.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.112568 12846 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
08:16:51.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.112671 12846 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.010s	user 0.000s	sys 0.001s
08:16:51.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.112776 12846 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
08:16:51.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.116035 12846 ts_tablet_manager.cc:616] Registered 1 tablets
08:16:51.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.116178 12846 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.001s	sys 0.001s
08:16:51.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.116787 12876 tablet_bootstrap.cc:492] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392: Bootstrap starting.
08:16:51.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.148548 12876 log.cc:826] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392: Log is configured to *not* fsync() on all Append() calls
08:16:51.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.188153 12846 rpc_server.cc:307] RPC server started. Bound to: 127.10.40.67:44983
08:16:51.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.188225 13008 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.40.67:44983 every 8 connection(s)
08:16:51.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.189812 12876 tablet_bootstrap.cc:492] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=21 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
08:16:51.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.190090 12846 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-2/data/info.pb
08:16:51.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.190559 12876 tablet_bootstrap.cc:492] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392: Bootstrap complete.
08:16:51.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.192435 12876 ts_tablet_manager.cc:1403] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392: Time spent bootstrapping tablet: real 0.076s	user 0.055s	sys 0.017s
08:16:51.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.194804 10401 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0lHOBl/build/asan/bin/kudu as pid 12846
08:16:51.195 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:16:51.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.208784 12876 raft_consensus.cc:359] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:51.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.210263 12876 raft_consensus.cc:740] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ea1996689fb4cf18b17a35790fba392, State: Initialized, Role: FOLLOWER
08:16:51.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.211472 12876 consensus_queue.cc:260] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 8.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:51.250 [DEBUG - kudu-nio-6] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 24 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644, tablet=b328961edc164a1289f259240593e73e, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:16:51.252 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.10.40.66:46247
08:16:51.253 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

08:16:51.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.253520 10401 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-task0lHOBl/build/asan/bin/kudu pid 12395
08:16:51.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.259054 13011 heartbeater.cc:344] Connected to a master server at 127.10.40.126:45793
08:16:51.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.259583 13011 heartbeater.cc:461] Registering TS with master...
08:16:51.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.261096 13011 heartbeater.cc:507] Master 127.10.40.126:45793 requested a full tablet report, sending...
08:16:51.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.265055 10451 ts_manager.cc:194] Re-registered known tserver with Master: 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67:44983)
08:16:51.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.268045 13009 heartbeater.cc:344] Connected to a master server at 127.10.40.124:41773
08:16:51.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.268218 13009 heartbeater.cc:461] Registering TS with master...
08:16:51.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.268904 13009 heartbeater.cc:507] Master 127.10.40.124:41773 requested a full tablet report, sending...
08:16:51.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.270538 10647 ts_manager.cc:194] Re-registered known tserver with Master: 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67:44983)
08:16:51.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.272598 10647 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.10.40.67:51771
08:16:51.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.283764 12553 generic_service.cc:196] Checking for leaks (request via RPC)
08:16:51.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.284073 13009 heartbeater.cc:499] Master 127.10.40.124:41773 was elected leader, sending a full tablet report...
08:16:51.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.292786 12876 ts_tablet_manager.cc:1434] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392: Time spent starting tablet: real 0.100s	user 0.040s	sys 0.041s
08:16:51.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.296017 13016 heartbeater.cc:344] Connected to a master server at 127.10.40.125:46051
08:16:51.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.296197 13016 heartbeater.cc:461] Registering TS with master...
08:16:51.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.296538 13016 heartbeater.cc:507] Master 127.10.40.125:46051 requested a full tablet report, sending...
08:16:51.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.298629 10549 ts_manager.cc:194] Re-registered known tserver with Master: 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67:44983)
08:16:51.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:51.403971 10401 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0lHOBl/build/asan/bin/kudu with pid 12395
08:16:51.431 [DEBUG - kudu-nio-6] (Connection.java:239) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] handling channelInactive
08:16:51.431 [DEBUG - kudu-nio-6] (Connection.java:709) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] cleaning up while in state READY due to: connection closed
08:16:51.432 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:16:51.432 [DEBUG - Time-limited test] (Connection.java:748) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] connecting to peer
08:16:51.433 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:51.433 [DEBUG - kudu-nio-7] (Connection.java:709) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] cleaning up while in state CONNECTING due to: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:51.434 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:51.446 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:51.446 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983),f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)[L],f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)]}]
08:16:51.447 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=2, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=14), Traces: [0ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [2ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [2ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [10ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [14ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@1649774854(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@56d6590e -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6b51cdff -> wakeup thread Time-limited test)) after lookup
08:16:51.447 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] connecting to peer
08:16:51.448 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:51.448 [DEBUG - kudu-nio-0] (Connection.java:709) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] cleaning up while in state CONNECTING due to: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:51.448 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:51.467 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:51.467 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983),f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)[L],f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)]}]
08:16:51.467 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=4, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=35), Traces: [0ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [2ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [2ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [10ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [14ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [15ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [16ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [16ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [34ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@1649774854(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@56d6590e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@927377804 after retry RPC@768159694), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6b51cdff -> wakeup thread Time-limited test -> (continuation of Deferred@927377804 after retry RPC@768159694))) after lookup
08:16:51.468 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] connecting to peer
08:16:51.469 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:51.469 [DEBUG - kudu-nio-1] (Connection.java:709) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] cleaning up while in state CONNECTING due to: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:51.470 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:51.506 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:51.507 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983),f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)[L],f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)]}]
08:16:51.508 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=6, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=75), Traces: [0ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [2ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [2ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [10ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [14ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [15ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [16ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [16ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [34ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [36ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [38ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [38ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [71ms] refreshing cache from master, [71ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [74ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@1649774854(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@56d6590e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@927377804 after retry RPC@768159694) -> (continuation of Deferred@316976084 after retry RPC@1991721241), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6b51cdff -> wakeup thread Time-limited test -> (continuation of Deferred@927377804 after retry RPC@768159694) -> (continuation of Deferred@316976084 after retry RPC@1991721241))) after lookup
08:16:51.508 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] connecting to peer
08:16:51.509 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:51.510 [DEBUG - kudu-nio-2] (Connection.java:709) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] cleaning up while in state CONNECTING due to: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:51.510 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:51.606 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:51.607 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983),f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)[L],f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)]}]
08:16:51.607 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=8, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=175), Traces: [0ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [2ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [2ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [10ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [14ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [15ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [16ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [16ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [34ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [36ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [38ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [38ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [71ms] refreshing cache from master, [71ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [74ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [76ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [78ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [78ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [171ms] refreshing cache from master, [171ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [174ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@1649774854(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@56d6590e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@927377804 after retry RPC@768159694) -> (continuation of Deferred@316976084 after retry RPC@1991721241) -> (continuation of Deferred@2133872338 after retry RPC@868444068), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6b51cdff -> wakeup thread Time-limited test -> (continuation of Deferred@927377804 after retry RPC@768159694) -> (continuation of Deferred@316976084 after retry RPC@1991721241) -> (continuation of Deferred@2133872338 after retry RPC@868444068))) after lookup
08:16:51.608 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] connecting to peer
08:16:51.609 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:51.609 [DEBUG - kudu-nio-3] (Connection.java:709) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] cleaning up while in state CONNECTING due to: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:51.609 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:51.706 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:51.707 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983),f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)[L],f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)]}]
08:16:51.707 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=10, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=275), Traces: [0ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [2ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [2ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [10ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [14ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [15ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [16ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [16ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [34ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [36ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [38ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [38ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [71ms] refreshing cache from master, [71ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [74ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [76ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [78ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [78ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [171ms] refreshing cache from master, [171ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [174ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [176ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [177ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [178ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [271ms] refreshing cache from master, [272ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [274ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@1649774854(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@56d6590e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@927377804 after retry RPC@768159694) -> (continuation of Deferred@316976084 after retry RPC@1991721241) -> (continuation of Deferred@2133872338 after retry RPC@868444068) -> (continuation of Deferred@1613949992 after retry RPC@1214532045), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6b51cdff -> wakeup thread Time-limited test -> (continuation of Deferred@927377804 after retry RPC@768159694) -> (continuation of Deferred@316976084 after retry RPC@1991721241) -> (continuation of Deferred@2133872338 after retry RPC@868444068) -> (continuation of Deferred@1613949992 after retry RPC@1214532045))) after lookup
08:16:51.708 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] connecting to peer
08:16:51.709 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:51.709 [DEBUG - kudu-nio-4] (Connection.java:709) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] cleaning up while in state CONNECTING due to: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:51.709 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:52.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:52.527902 13029 raft_consensus.cc:493] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f0e29bd27cb448589ac37e569ad1a164)
08:16:52.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:52.528213 13029 raft_consensus.cc:515] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:52.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:52.529801 13029 leader_election.cc:290] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247), 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67:44983)
08:16:52.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:52.531221 12648 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.40.66:46247: connect: Connection refused (error 111)
08:16:52.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:52.533224 12648 leader_election.cc:336] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247): Network error: Client connection negotiation failed: client connection to 127.10.40.66:46247: connect: Connection refused (error 111)
08:16:52.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:52.540158 12955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b328961edc164a1289f259240593e73e" candidate_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" candidate_term: 10 candidate_status { last_received { term: 9 index: 32 } } ignore_live_leader: false dest_uuid: "2ea1996689fb4cf18b17a35790fba392" is_pre_election: true
08:16:52.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:52.540723 12955 raft_consensus.cc:2468] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2a3f45a9e3b4f2dbda7e2613df89453 in term 8.
08:16:52.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:52.541628 12648 leader_election.cc:304] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ea1996689fb4cf18b17a35790fba392, f2a3f45a9e3b4f2dbda7e2613df89453; no voters: f0e29bd27cb448589ac37e569ad1a164
08:16:52.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:52.541991 13029 raft_consensus.cc:2804] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 9 FOLLOWER]: Leader pre-election won for term 10
08:16:52.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:52.542131 13029 raft_consensus.cc:493] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 9 FOLLOWER]: Starting leader election (detected failure of leader f0e29bd27cb448589ac37e569ad1a164)
08:16:52.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:52.542249 13029 raft_consensus.cc:3060] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 9 FOLLOWER]: Advancing to term 10
08:16:52.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:52.544611 13029 raft_consensus.cc:515] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:52.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:52.545228 13029 leader_election.cc:290] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [CANDIDATE]: Term 10 election: Requested vote from peers f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247), 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67:44983)
08:16:52.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:52.546167 12955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b328961edc164a1289f259240593e73e" candidate_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" candidate_term: 10 candidate_status { last_received { term: 9 index: 32 } } ignore_live_leader: false dest_uuid: "2ea1996689fb4cf18b17a35790fba392"
08:16:52.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:52.546463 12955 raft_consensus.cc:3060] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 8 FOLLOWER]: Advancing to term 10
08:16:52.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:52.546928 12648 leader_election.cc:336] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247): Network error: Client connection negotiation failed: client connection to 127.10.40.66:46247: connect: Connection refused (error 111)
08:16:52.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:52.551043 12955 raft_consensus.cc:2468] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2a3f45a9e3b4f2dbda7e2613df89453 in term 10.
08:16:52.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:52.551577 12648 leader_election.cc:304] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ea1996689fb4cf18b17a35790fba392, f2a3f45a9e3b4f2dbda7e2613df89453; no voters: f0e29bd27cb448589ac37e569ad1a164
08:16:52.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:52.551889 13029 raft_consensus.cc:2804] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 10 FOLLOWER]: Leader election won for term 10
08:16:52.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:52.552412 13029 raft_consensus.cc:697] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 10 LEADER]: Becoming Leader. State: Replica: f2a3f45a9e3b4f2dbda7e2613df89453, State: Running, Role: LEADER
08:16:52.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:52.552984 13029 consensus_queue.cc:237] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 9.32, Last appended by leader: 32, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:52.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:52.556699 10647 catalog_manager.cc:5719] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 reported cstate change: term changed from 9 to 10, leader changed from f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66) to f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65). New cstate: current_term: 10 leader_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } health_report { overall_health: HEALTHY } } }
08:16:53.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:53.049355 12955 raft_consensus.cc:1275] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 10 FOLLOWER]: Refusing update from remote peer f2a3f45a9e3b4f2dbda7e2613df89453: Log matching property violated. Preceding OpId in replica: term: 8 index: 28. Preceding OpId from leader: term: 10 index: 33. (index mismatch)
08:16:53.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:53.050527 13029 consensus_queue.cc:1048] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 28, Time since last communication: 0.000s
08:16:53.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:53.066932 12648 consensus_peers.cc:597] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 -> Peer f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247): Couldn't send request to peer f0e29bd27cb448589ac37e569ad1a164. Status: Network error: Client connection negotiation failed: client connection to 127.10.40.66:46247: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
08:16:53.486 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:53.487 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983),f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247),f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)[L]]}]
08:16:53.487 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=12, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=2055), Traces: [0ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [2ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [2ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [10ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [14ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [15ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [16ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [16ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [34ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [36ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [38ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [38ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [71ms] refreshing cache from master, [71ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [74ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [76ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [78ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [78ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [171ms] refreshing cache from master, [171ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [174ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [176ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [177ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [178ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [271ms] refreshing cache from master, [272ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [274ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [275ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [277ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [277ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [2051ms] refreshing cache from master, [2051ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [2054ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@1649774854(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@56d6590e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@927377804 after retry RPC@768159694) -> (continuation of Deferred@316976084 after retry RPC@1991721241) -> (continuation of Deferred@2133872338 after retry RPC@868444068) -> (continuation of Deferred@1613949992 after retry RPC@1214532045) -> (continuation of Deferred@1308295309 after retry RPC@666567390), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6b51cdff -> wakeup thread Time-limited test -> (continuation of Deferred@927377804 after retry RPC@768159694) -> (continuation of Deferred@316976084 after retry RPC@1991721241) -> (continuation of Deferred@2133872338 after retry RPC@868444068) -> (continuation of Deferred@1613949992 after retry RPC@1214532045) -> (continuation of Deferred@1308295309 after retry RPC@666567390))) after lookup
08:16:53.488 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] connecting to peer
08:16:53.489 [DEBUG - kudu-nio-5] (Connection.java:757) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] Successfully connected to peer
08:16:53.521 [DEBUG - kudu-nio-5] (Negotiator.java:918) Authenticated connection [id: 0x4530bd67, L:/127.0.0.1:54612 - R:/127.10.40.65:39249] using TOKEN/null
08:16:53.547 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.10.40.66:46247
08:16:53.547 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

08:16:53.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:53.548098 10401 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0lHOBl/build/asan/bin/kudu
08:16:53.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task0lHOBl/build/asan/bin/kudu
08:16:53.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/wal
08:16:53.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/data
08:16:53.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:16:53.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:16:53.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:16:53.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:16:53.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:16:53.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:16:53.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/logs
08:16:53.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/data/info.pb
08:16:53.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:16:53.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:16:53.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:16:53.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:16:53.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:16:53.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:16:53.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:16:53.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:16:53.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:16:53.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:16:53.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.40.66:46247
08:16:53.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.40.66
08:16:53.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.40.126:45793,127.10.40.125:46051,127.10.40.124:41773
08:16:53.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=33605
08:16:53.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.40.66
08:16:53.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.40.84:40955
08:16:53.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:16:53.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:16:53.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:16:53.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:53.905035 13069 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:16:53.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:53.905378 13069 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:16:53.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:53.905467 13069 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:16:53.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:53.914041 13069 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:16:53.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:53.914216 13069 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.40.66
08:16:53.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:53.924458 13069 tablet_server_runner.cc:78] Tablet server non-default flags:
08:16:53.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/logs
08:16:53.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:16:53.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:16:53.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:16:53.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.40.84:40955
08:16:53.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:16:53.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:16:53.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/data
08:16:53.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/wal
08:16:53.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:16:53.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:16:53.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.40.66:46247
08:16:53.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:16:53.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:16:53.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:16:53.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/data/info.pb
08:16:53.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.40.66
08:16:53.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=33605
08:16:53.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.40.126:45793,127.10.40.125:46051,127.10.40.124:41773
08:16:53.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:16:53.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:16:53.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:16:53.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:16:53.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:16:53.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.40.66
08:16:53.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:16:53.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:16:53.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:16:53.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 70c51312f1186a8ae9f46647160b7730123f7601
08:16:53.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:16:53.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 12 Aug 2026 07:43:21 UTC on bdcb31816ec0
08:16:53.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 13302
08:16:53.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
08:16:53.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:53.926235 13069 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:16:53.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:53.928212 13069 file_cache.cc:504] Constructed file cache file cache with capacity 419430
08:16:53.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:53.938954 13076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:16:53.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:53.939325 13077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:16:53.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:53.939683 13080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:16:53.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:53.940325 13069 server_base.cc:1061] running on GCE node
08:16:53.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:53.940950 13069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:16:53.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:53.941927 13069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:16:53.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:53.943141 13069 hybrid_clock.cc:648] HybridClock initialized: now 1786522613943075 us; error 62 us; skew 500 ppm
08:16:53.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:53.943424 13069 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:16:53.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:53.946556 13069 webserver.cc:533] Webserver started at http://127.10.40.66:33605/ using document root <none> and password file <none>
08:16:53.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:53.947365 13069 fs_manager.cc:362] Metadata directory not provided
08:16:53.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:53.947517 13069 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:16:53.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:53.953261 13069 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.006s	sys 0.000s
08:16:53.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:53.956370 13087 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:16:53.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:53.957688 13069 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
08:16:53.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:53.957898 13069 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/data,/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/wal
08:16:53.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "f0e29bd27cb448589ac37e569ad1a164"
08:16:53.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:16:19 on dist-test-slave-gp6n"
08:16:53.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:53.958760 13069 fs_report.cc:389] FS layout report
08:16:53.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:16:53.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/wal
08:16:53.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/wal
08:16:53.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/data/data
08:16:53.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:16:53.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:16:53.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:16:53.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:16:53.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:16:53.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:16:53.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:16:53.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:16:53.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:16:53.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:16:53.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:16:53.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:53.981451 13069 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:16:53.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:53.982542 13069 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:16:53.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:53.982923 13069 kserver.cc:163] Server-wide thread pool size limit: 3276
08:16:53.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:53.984102 13069 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
08:16:53.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:53.986773 13096 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
08:16:53.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:53.993692 13069 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
08:16:53.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:53.993817 13069 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.007s	user 0.000s	sys 0.001s
08:16:53.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:53.993978 13069 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
08:16:53.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:53.997242 13069 ts_tablet_manager.cc:616] Registered 1 tablets
08:16:53.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:53.997388 13069 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.000s	sys 0.001s
08:16:53.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:53.997890 13096 tablet_bootstrap.cc:492] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164: Bootstrap starting.
08:16:54.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:54.027832 13096 log.cc:826] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164: Log is configured to *not* fsync() on all Append() calls
08:16:54.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:54.064262 13069 rpc_server.cc:307] RPC server started. Bound to: 127.10.40.66:46247
08:16:54.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:54.064312 13236 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.40.66:46247 every 8 connection(s)
08:16:54.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:54.066682 13069 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/data/info.pb
08:16:54.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:54.073984 10401 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0lHOBl/build/asan/bin/kudu as pid 13069
08:16:54.074 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:16:54.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:54.091238 13096 tablet_bootstrap.cc:492] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=24 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
08:16:54.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:54.092212 13096 tablet_bootstrap.cc:492] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164: Bootstrap complete.
08:16:54.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:54.094864 13096 ts_tablet_manager.cc:1403] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164: Time spent bootstrapping tablet: real 0.097s	user 0.065s	sys 0.020s
08:16:54.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:54.106331 13096 raft_consensus.cc:359] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 9 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:54.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:54.107555 13096 raft_consensus.cc:740] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e29bd27cb448589ac37e569ad1a164, State: Initialized, Role: FOLLOWER
08:16:54.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:54.108349 13096 consensus_queue.cc:260] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 9.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:54.115 [DEBUG - kudu-nio-5] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 27 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644, tablet=b328961edc164a1289f259240593e73e, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:16:54.119 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.10.40.65:39249
08:16:54.119 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

08:16:54.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:54.119982 10401 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-task0lHOBl/build/asan/bin/kudu pid 12631
08:16:54.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:54.121711 13238 heartbeater.cc:344] Connected to a master server at 127.10.40.126:45793
08:16:54.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:54.122200 13238 heartbeater.cc:461] Registering TS with master...
08:16:54.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:54.123448 13238 heartbeater.cc:507] Master 127.10.40.126:45793 requested a full tablet report, sending...
08:16:54.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:54.129125 13237 heartbeater.cc:344] Connected to a master server at 127.10.40.124:41773
08:16:54.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:54.129329 13237 heartbeater.cc:461] Registering TS with master...
08:16:54.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:54.129980 13237 heartbeater.cc:507] Master 127.10.40.124:41773 requested a full tablet report, sending...
08:16:54.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:54.130584 10451 ts_manager.cc:194] Re-registered known tserver with Master: f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247)
08:16:54.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:54.131577 10647 ts_manager.cc:194] Re-registered known tserver with Master: f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247)
08:16:54.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:54.133210 10647 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.10.40.66:50259
08:16:54.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:54.152360 13244 heartbeater.cc:344] Connected to a master server at 127.10.40.125:46051
08:16:54.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:54.152582 13244 heartbeater.cc:461] Registering TS with master...
08:16:54.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:54.153085 13244 heartbeater.cc:507] Master 127.10.40.125:46051 requested a full tablet report, sending...
08:16:54.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:54.154850 10549 ts_manager.cc:194] Re-registered known tserver with Master: f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247)
08:16:54.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:54.156775 13237 heartbeater.cc:499] Master 127.10.40.124:41773 was elected leader, sending a full tablet report...
08:16:54.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:54.157717 13096 ts_tablet_manager.cc:1434] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164: Time spent starting tablet: real 0.063s	user 0.020s	sys 0.016s
08:16:54.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:54.170037 13175 raft_consensus.cc:3060] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 9 FOLLOWER]: Advancing to term 10
08:16:54.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:54.186949 12794 generic_service.cc:196] Checking for leaks (request via RPC)
08:16:54.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:54.189488 13256 mvcc.cc:204] Tried to move back new op lower bound from 7317596624995794944 to 7317596621020168192. Current Snapshot: MvccSnapshot[applied={T|T < 7317596624995794944}]
08:16:54.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:54.346844 10401 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0lHOBl/build/asan/bin/kudu with pid 12631
08:16:54.373 [DEBUG - kudu-nio-5] (Connection.java:239) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] handling channelInactive
08:16:54.374 [DEBUG - kudu-nio-5] (Connection.java:709) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] cleaning up while in state READY due to: connection closed
08:16:54.374 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:16:54.375 [DEBUG - Time-limited test] (Connection.java:748) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] connecting to peer
08:16:54.376 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:54.376 [DEBUG - kudu-nio-6] (Connection.java:709) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] cleaning up while in state CONNECTING due to: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:54.376 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:54.387 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:54.387 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983),f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247),f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)[L]]}]
08:16:54.388 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=2, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=12), Traces: [0ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [1ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [1ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [12ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@2143641397(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@970fe1 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@12586f8f -> wakeup thread Time-limited test)) after lookup
08:16:54.388 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] connecting to peer
08:16:54.390 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:54.390 [DEBUG - kudu-nio-7] (Connection.java:709) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] cleaning up while in state CONNECTING due to: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:54.390 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:54.406 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:54.407 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983),f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247),f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)[L]]}]
08:16:54.407 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=4, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=32), Traces: [0ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [1ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [1ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [12ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [13ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [15ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [15ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [31ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@2143641397(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@970fe1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1802943797 after retry RPC@981701571), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@12586f8f -> wakeup thread Time-limited test -> (continuation of Deferred@1802943797 after retry RPC@981701571))) after lookup
08:16:54.408 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] connecting to peer
08:16:54.409 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:54.410 [DEBUG - kudu-nio-0] (Connection.java:709) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] cleaning up while in state CONNECTING due to: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:54.410 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:54.426 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:54.426 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983),f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247),f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)[L]]}]
08:16:54.427 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=6, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=51), Traces: [0ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [1ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [1ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [12ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [13ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [15ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [15ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [31ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [33ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [35ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [35ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [48ms] refreshing cache from master, [48ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [51ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@2143641397(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@970fe1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1802943797 after retry RPC@981701571) -> (continuation of Deferred@1154479629 after retry RPC@746224980), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@12586f8f -> wakeup thread Time-limited test -> (continuation of Deferred@1802943797 after retry RPC@981701571) -> (continuation of Deferred@1154479629 after retry RPC@746224980))) after lookup
08:16:54.428 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] connecting to peer
08:16:54.429 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:54.429 [DEBUG - kudu-nio-1] (Connection.java:709) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] cleaning up while in state CONNECTING due to: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:54.429 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:54.507 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:54.507 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983),f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247),f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)[L]]}]
08:16:54.507 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=8, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=132), Traces: [0ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [1ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [1ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [12ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [13ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [15ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [15ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [31ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [33ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [35ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [35ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [48ms] refreshing cache from master, [48ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [51ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [52ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [54ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [54ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [128ms] refreshing cache from master, [128ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [131ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@2143641397(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@970fe1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1802943797 after retry RPC@981701571) -> (continuation of Deferred@1154479629 after retry RPC@746224980) -> (continuation of Deferred@1840404167 after retry RPC@308086520), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@12586f8f -> wakeup thread Time-limited test -> (continuation of Deferred@1802943797 after retry RPC@981701571) -> (continuation of Deferred@1154479629 after retry RPC@746224980) -> (continuation of Deferred@1840404167 after retry RPC@308086520))) after lookup
08:16:54.508 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] connecting to peer
08:16:54.509 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:54.509 [DEBUG - kudu-nio-2] (Connection.java:709) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] cleaning up while in state CONNECTING due to: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:54.510 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:54.586 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:54.587 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983),f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247),f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)[L]]}]
08:16:54.587 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=10, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=212), Traces: [0ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [1ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [1ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [12ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [13ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [15ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [15ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [31ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [33ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [35ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [35ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [48ms] refreshing cache from master, [48ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [51ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [52ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [54ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [54ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [128ms] refreshing cache from master, [128ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [131ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [133ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [135ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [135ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [208ms] refreshing cache from master, [208ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [211ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@2143641397(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@970fe1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1802943797 after retry RPC@981701571) -> (continuation of Deferred@1154479629 after retry RPC@746224980) -> (continuation of Deferred@1840404167 after retry RPC@308086520) -> (continuation of Deferred@2145906710 after retry RPC@100090836), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@12586f8f -> wakeup thread Time-limited test -> (continuation of Deferred@1802943797 after retry RPC@981701571) -> (continuation of Deferred@1154479629 after retry RPC@746224980) -> (continuation of Deferred@1840404167 after retry RPC@308086520) -> (continuation of Deferred@2145906710 after retry RPC@100090836))) after lookup
08:16:54.587 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] connecting to peer
08:16:54.588 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:54.588 [DEBUG - kudu-nio-3] (Connection.java:709) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] cleaning up while in state CONNECTING due to: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:54.588 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:55.486 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:55.487 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983),f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247),f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)[L]]}]
08:16:55.487 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=12, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=1112), Traces: [0ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [1ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [1ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [12ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [13ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [15ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [15ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [31ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [33ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [35ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [35ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [48ms] refreshing cache from master, [48ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [51ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [52ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [54ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [54ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [128ms] refreshing cache from master, [128ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [131ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [133ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [135ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [135ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [208ms] refreshing cache from master, [208ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [211ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [212ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [213ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [213ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [1108ms] refreshing cache from master, [1108ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [1111ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@2143641397(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@970fe1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1802943797 after retry RPC@981701571) -> (continuation of Deferred@1154479629 after retry RPC@746224980) -> (continuation of Deferred@1840404167 after retry RPC@308086520) -> (continuation of Deferred@2145906710 after retry RPC@100090836) -> (continuation of Deferred@1775861377 after retry RPC@1443854733), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@12586f8f -> wakeup thread Time-limited test -> (continuation of Deferred@1802943797 after retry RPC@981701571) -> (continuation of Deferred@1154479629 after retry RPC@746224980) -> (continuation of Deferred@1840404167 after retry RPC@308086520) -> (continuation of Deferred@2145906710 after retry RPC@100090836) -> (continuation of Deferred@1775861377 after retry RPC@1443854733))) after lookup
08:16:55.488 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] connecting to peer
08:16:55.489 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:55.489 [DEBUG - kudu-nio-4] (Connection.java:709) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] cleaning up while in state CONNECTING due to: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:55.489 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249
08:16:55.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:55.566170 13265 raft_consensus.cc:493] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f2a3f45a9e3b4f2dbda7e2613df89453)
08:16:55.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:55.566461 13265 raft_consensus.cc:515] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:55.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:55.568344 13265 leader_election.cc:290] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247), f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65:39249)
08:16:55.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:55.570905 12869 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.40.65:39249: connect: Connection refused (error 111)
08:16:55.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:55.573551 12869 leader_election.cc:336] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65:39249): Network error: Client connection negotiation failed: client connection to 127.10.40.65:39249: connect: Connection refused (error 111)
08:16:55.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:55.575798 13175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b328961edc164a1289f259240593e73e" candidate_uuid: "2ea1996689fb4cf18b17a35790fba392" candidate_term: 11 candidate_status { last_received { term: 10 index: 36 } } ignore_live_leader: false dest_uuid: "f0e29bd27cb448589ac37e569ad1a164" is_pre_election: true
08:16:55.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:55.576841 12869 leader_election.cc:304] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ea1996689fb4cf18b17a35790fba392; no voters: f0e29bd27cb448589ac37e569ad1a164, f2a3f45a9e3b4f2dbda7e2613df89453
08:16:55.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:55.577164 13265 raft_consensus.cc:2749] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
08:16:55.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:55.675774 13271 raft_consensus.cc:493] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f2a3f45a9e3b4f2dbda7e2613df89453)
08:16:55.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:55.676052 13271 raft_consensus.cc:515] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:55.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:55.677541 13271 leader_election.cc:290] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67:44983), f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65:39249)
08:16:55.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:55.679600 13088 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.40.65:39249: connect: Connection refused (error 111)
08:16:55.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:55.681100 13088 leader_election.cc:336] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65:39249): Network error: Client connection negotiation failed: client connection to 127.10.40.65:39249: connect: Connection refused (error 111)
08:16:55.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:55.684353 12955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b328961edc164a1289f259240593e73e" candidate_uuid: "f0e29bd27cb448589ac37e569ad1a164" candidate_term: 11 candidate_status { last_received { term: 10 index: 36 } } ignore_live_leader: false dest_uuid: "2ea1996689fb4cf18b17a35790fba392" is_pre_election: true
08:16:55.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:55.684576 12955 raft_consensus.cc:2468] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0e29bd27cb448589ac37e569ad1a164 in term 10.
08:16:55.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:55.685102 13088 leader_election.cc:304] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ea1996689fb4cf18b17a35790fba392, f0e29bd27cb448589ac37e569ad1a164; no voters: f2a3f45a9e3b4f2dbda7e2613df89453
08:16:55.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:55.685536 13271 raft_consensus.cc:2804] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 10 FOLLOWER]: Leader pre-election won for term 11
08:16:55.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:55.685652 13271 raft_consensus.cc:493] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 10 FOLLOWER]: Starting leader election (detected failure of leader f2a3f45a9e3b4f2dbda7e2613df89453)
08:16:55.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:55.685719 13271 raft_consensus.cc:3060] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 10 FOLLOWER]: Advancing to term 11
08:16:55.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:55.687860 13271 raft_consensus.cc:515] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:55.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:55.688355 13271 leader_election.cc:290] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [CANDIDATE]: Term 11 election: Requested vote from peers 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67:44983), f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65:39249)
08:16:55.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:55.688934 12955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b328961edc164a1289f259240593e73e" candidate_uuid: "f0e29bd27cb448589ac37e569ad1a164" candidate_term: 11 candidate_status { last_received { term: 10 index: 36 } } ignore_live_leader: false dest_uuid: "2ea1996689fb4cf18b17a35790fba392"
08:16:55.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:55.689164 12955 raft_consensus.cc:3060] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 10 FOLLOWER]: Advancing to term 11
08:16:55.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:55.689967 13088 leader_election.cc:336] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65:39249): Network error: Client connection negotiation failed: client connection to 127.10.40.65:39249: connect: Connection refused (error 111)
08:16:55.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:55.691637 12955 raft_consensus.cc:2468] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0e29bd27cb448589ac37e569ad1a164 in term 11.
08:16:55.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:55.692124 13088 leader_election.cc:304] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ea1996689fb4cf18b17a35790fba392, f0e29bd27cb448589ac37e569ad1a164; no voters: f2a3f45a9e3b4f2dbda7e2613df89453
08:16:55.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:55.692471 13271 raft_consensus.cc:2804] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 11 FOLLOWER]: Leader election won for term 11
08:16:55.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:55.692858 13271 raft_consensus.cc:697] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 11 LEADER]: Becoming Leader. State: Replica: f0e29bd27cb448589ac37e569ad1a164, State: Running, Role: LEADER
08:16:55.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:55.693395 13271 consensus_queue.cc:237] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 10.36, Last appended by leader: 36, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:55.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:55.697101 10647 catalog_manager.cc:5719] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 reported cstate change: term changed from 10 to 11, leader changed from f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65) to f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66). New cstate: current_term: 11 leader_uuid: "f0e29bd27cb448589ac37e569ad1a164" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } health_report { overall_health: UNKNOWN } } }
08:16:56.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:56.127199 12955 raft_consensus.cc:1275] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 11 FOLLOWER]: Refusing update from remote peer f0e29bd27cb448589ac37e569ad1a164: Log matching property violated. Preceding OpId in replica: term: 10 index: 36. Preceding OpId from leader: term: 11 index: 37. (index mismatch)
08:16:56.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:56.128188 13271 consensus_queue.cc:1048] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
08:16:56.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:56.135175 13088 consensus_peers.cc:597] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 -> Peer f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65:39249): Couldn't send request to peer f2a3f45a9e3b4f2dbda7e2613df89453. Status: Network error: Client connection negotiation failed: client connection to 127.10.40.65:39249: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
08:16:58.686 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:58.687 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983),f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)[L],f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)]}]
08:16:58.687 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=14, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=4312), Traces: [0ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [1ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [1ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [12ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [13ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [15ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [15ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [31ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [33ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [35ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [35ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [48ms] refreshing cache from master, [48ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [51ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [52ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [54ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [54ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [128ms] refreshing cache from master, [128ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [131ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [133ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [135ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [135ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [208ms] refreshing cache from master, [208ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [211ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [212ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [213ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [213ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [1108ms] refreshing cache from master, [1108ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [1111ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [1113ms] sending RPC to server f2a3f45a9e3b4f2dbda7e2613df89453, [1114ms] received response from server f2a3f45a9e3b4f2dbda7e2613df89453: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [1114ms] delaying RPC due to: Network error: Failed to connect to peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249): Connection refused: /127.10.40.65:39249, [4308ms] refreshing cache from master, [4308ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [4311ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@2143641397(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@970fe1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1802943797 after retry RPC@981701571) -> (continuation of Deferred@1154479629 after retry RPC@746224980) -> (continuation of Deferred@1840404167 after retry RPC@308086520) -> (continuation of Deferred@2145906710 after retry RPC@100090836) -> (continuation of Deferred@1775861377 after retry RPC@1443854733) -> (continuation of Deferred@2046264309 after retry RPC@1410944599), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@12586f8f -> wakeup thread Time-limited test -> (continuation of Deferred@1802943797 after retry RPC@981701571) -> (continuation of Deferred@1154479629 after retry RPC@746224980) -> (continuation of Deferred@1840404167 after retry RPC@308086520) -> (continuation of Deferred@2145906710 after retry RPC@100090836) -> (continuation of Deferred@1775861377 after retry RPC@1443854733) -> (continuation of Deferred@2046264309 after retry RPC@1410944599))) after lookup
08:16:58.688 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] connecting to peer
08:16:58.688 [DEBUG - kudu-nio-5] (Connection.java:757) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] Successfully connected to peer
08:16:58.703 [DEBUG - kudu-nio-5] (Negotiator.java:918) Authenticated connection [id: 0x9cc60c42, L:/127.0.0.1:44706 - R:/127.10.40.66:46247] using TOKEN/null
08:16:58.728 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.10.40.65:39249
08:16:58.729 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

08:16:58.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:58.729854 10401 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0lHOBl/build/asan/bin/kudu
08:16:58.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task0lHOBl/build/asan/bin/kudu
08:16:58.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/wal
08:16:58.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/data
08:16:58.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:16:58.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:16:58.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:16:58.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:16:58.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:16:58.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:16:58.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/logs
08:16:58.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/data/info.pb
08:16:58.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:16:58.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:16:58.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:16:58.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:16:58.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:16:58.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:16:58.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:16:58.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:16:58.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:16:58.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:16:58.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.40.65:39249
08:16:58.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.40.65
08:16:58.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.40.126:45793,127.10.40.125:46051,127.10.40.124:41773
08:16:58.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=41939
08:16:58.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.40.65
08:16:58.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.40.84:40955
08:16:58.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:16:58.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:16:58.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:16:58.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:58.755147 13088 consensus_peers.cc:597] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 -> Peer f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65:39249): Couldn't send request to peer f2a3f45a9e3b4f2dbda7e2613df89453. Status: Network error: Client connection negotiation failed: client connection to 127.10.40.65:39249: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
08:16:59.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:59.091442 13319 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:16:59.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:59.091760 13319 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:16:59.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:59.091836 13319 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:16:59.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:59.100641 13319 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:16:59.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:59.100816 13319 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.40.65
08:16:59.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.111027 13319 tablet_server_runner.cc:78] Tablet server non-default flags:
08:16:59.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/logs
08:16:59.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:16:59.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:16:59.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:16:59.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.40.84:40955
08:16:59.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:16:59.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:16:59.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/data
08:16:59.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/wal
08:16:59.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:16:59.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:16:59.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.40.65:39249
08:16:59.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:16:59.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:16:59.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:16:59.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/data/info.pb
08:16:59.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.40.65
08:16:59.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=41939
08:16:59.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.40.126:45793,127.10.40.125:46051,127.10.40.124:41773
08:16:59.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:16:59.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:16:59.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:16:59.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:16:59.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:16:59.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.40.65
08:16:59.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:16:59.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:16:59.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:16:59.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 70c51312f1186a8ae9f46647160b7730123f7601
08:16:59.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:16:59.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 12 Aug 2026 07:43:21 UTC on bdcb31816ec0
08:16:59.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 13302
08:16:59.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
08:16:59.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.112758 13319 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:16:59.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.114704 13319 file_cache.cc:504] Constructed file cache file cache with capacity 419430
08:16:59.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:59.124766 13324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:16:59.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:59.124796 13325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:16:59.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:16:59.126077 13327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:16:59.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.126950 13319 server_base.cc:1061] running on GCE node
08:16:59.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.127472 13319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:16:59.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.128448 13319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:16:59.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.129659 13319 hybrid_clock.cc:648] HybridClock initialized: now 1786522619129586 us; error 65 us; skew 500 ppm
08:16:59.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.129943 13319 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:16:59.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.132172 13319 webserver.cc:533] Webserver started at http://127.10.40.65:41939/ using document root <none> and password file <none>
08:16:59.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.132880 13319 fs_manager.cc:362] Metadata directory not provided
08:16:59.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.132972 13319 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:16:59.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.138258 13319 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.002s	sys 0.000s
08:16:59.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.141147 13336 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:16:59.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.142642 13319 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
08:16:59.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.142987 13319 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/data,/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/wal
08:16:59.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "f2a3f45a9e3b4f2dbda7e2613df89453"
08:16:59.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:16:19 on dist-test-slave-gp6n"
08:16:59.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.144223 13319 fs_report.cc:389] FS layout report
08:16:59.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:16:59.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/wal
08:16:59.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/wal
08:16:59.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/data/data
08:16:59.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:16:59.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:16:59.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:16:59.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:16:59.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:16:59.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:16:59.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:16:59.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:16:59.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:16:59.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:16:59.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:16:59.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.167915 13319 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:16:59.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.168731 13319 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:16:59.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.168994 13319 kserver.cc:163] Server-wide thread pool size limit: 3276
08:16:59.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.170203 13319 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
08:16:59.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.173024 13343 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
08:16:59.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.179342 13319 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
08:16:59.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.179437 13319 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.007s	user 0.000s	sys 0.001s
08:16:59.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.179561 13319 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
08:16:59.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.182723 13319 ts_tablet_manager.cc:616] Registered 1 tablets
08:16:59.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.182818 13319 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.003s	sys 0.000s
08:16:59.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.183341 13343 tablet_bootstrap.cc:492] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453: Bootstrap starting.
08:16:59.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.218256 13343 log.cc:826] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453: Log is configured to *not* fsync() on all Append() calls
08:16:59.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.231423 13319 rpc_server.cc:307] RPC server started. Bound to: 127.10.40.65:39249
08:16:59.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.231463 13483 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.40.65:39249 every 8 connection(s)
08:16:59.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.233191 13319 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-0/data/info.pb
08:16:59.236 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:16:59.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.236287 10401 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0lHOBl/build/asan/bin/kudu as pid 13319
08:16:59.279 [DEBUG - kudu-nio-5] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 30 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644, tablet=b328961edc164a1289f259240593e73e, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:16:59.280 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.10.40.66:46247
08:16:59.280 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

08:16:59.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.281173 10401 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-task0lHOBl/build/asan/bin/kudu pid 13069
08:16:59.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.292268 13343 tablet_bootstrap.cc:492] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453: Bootstrap replayed 1/1 log segments. Stats: ops{read=36 overwritten=0 applied=36 ignored=0} inserts{seen=27 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
08:16:59.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.293191 13343 tablet_bootstrap.cc:492] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453: Bootstrap complete.
08:16:59.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.297118 13343 ts_tablet_manager.cc:1403] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453: Time spent bootstrapping tablet: real 0.114s	user 0.079s	sys 0.015s
08:16:59.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.304327 13343 raft_consensus.cc:359] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 10 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:59.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.305122 13343 raft_consensus.cc:740] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 10 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2a3f45a9e3b4f2dbda7e2613df89453, State: Initialized, Role: FOLLOWER
08:16:59.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.305846 13343 consensus_queue.cc:260] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 10.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:16:59.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.307785 13343 ts_tablet_manager.cc:1434] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.004s
08:16:59.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.310616 13485 heartbeater.cc:344] Connected to a master server at 127.10.40.126:45793
08:16:59.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.310732 13488 heartbeater.cc:344] Connected to a master server at 127.10.40.125:46051
08:16:59.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.311075 13488 heartbeater.cc:461] Registering TS with master...
08:16:59.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.311082 13485 heartbeater.cc:461] Registering TS with master...
08:16:59.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.311134 13229 generic_service.cc:196] Checking for leaks (request via RPC)
08:16:59.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.312184 13484 heartbeater.cc:344] Connected to a master server at 127.10.40.124:41773
08:16:59.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.312316 13484 heartbeater.cc:461] Registering TS with master...
08:16:59.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.312451 13488 heartbeater.cc:507] Master 127.10.40.125:46051 requested a full tablet report, sending...
08:16:59.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.315583 13485 heartbeater.cc:507] Master 127.10.40.126:45793 requested a full tablet report, sending...
08:16:59.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.316825 13484 heartbeater.cc:507] Master 127.10.40.124:41773 requested a full tablet report, sending...
08:16:59.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.317158 10549 ts_manager.cc:194] Re-registered known tserver with Master: f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65:39249)
08:16:59.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.318359 10647 ts_manager.cc:194] Re-registered known tserver with Master: f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65:39249)
08:16:59.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.320078 10647 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.10.40.65:36687
08:16:59.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.323308 10451 ts_manager.cc:194] Re-registered known tserver with Master: f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65:39249)
08:16:59.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.323554 13484 heartbeater.cc:499] Master 127.10.40.124:41773 was elected leader, sending a full tablet report...
08:16:59.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:16:59.425805 10401 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0lHOBl/build/asan/bin/kudu with pid 13069
08:16:59.461 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:16:59.461 [DEBUG - kudu-nio-5] (Connection.java:239) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] handling channelInactive
08:16:59.461 [DEBUG - kudu-nio-5] (Connection.java:709) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] cleaning up while in state READY due to: connection closed
08:16:59.462 [DEBUG - Time-limited test] (Connection.java:748) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] connecting to peer
08:16:59.463 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:59.463 [DEBUG - kudu-nio-6] (Connection.java:709) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] cleaning up while in state CONNECTING due to: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:59.463 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:59.486 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:59.487 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983),f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)[L],f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)]}]
08:16:59.487 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=2, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=25), Traces: [0ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [2ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [2ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [25ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@1578262747(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@4067e6b8 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@49ec74c0 -> wakeup thread Time-limited test)) after lookup
08:16:59.487 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] connecting to peer
08:16:59.488 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:59.489 [DEBUG - kudu-nio-7] (Connection.java:709) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] cleaning up while in state CONNECTING due to: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:59.489 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:59.507 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:59.507 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983),f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)[L],f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)]}]
08:16:59.508 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=4, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=46), Traces: [0ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [2ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [2ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [25ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [26ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [28ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [28ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [42ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [46ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@1578262747(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@4067e6b8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@490905849 after retry RPC@933404133), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@49ec74c0 -> wakeup thread Time-limited test -> (continuation of Deferred@490905849 after retry RPC@933404133))) after lookup
08:16:59.509 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] connecting to peer
08:16:59.510 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:59.510 [DEBUG - kudu-nio-0] (Connection.java:709) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] cleaning up while in state CONNECTING due to: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:59.510 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:59.547 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:59.547 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983),f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)[L],f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)]}]
08:16:59.547 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=6, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=86), Traces: [0ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [2ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [2ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [25ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [26ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [28ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [28ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [42ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [46ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [47ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [49ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [49ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [82ms] refreshing cache from master, [82ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [85ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@1578262747(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@4067e6b8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@490905849 after retry RPC@933404133) -> (continuation of Deferred@450532377 after retry RPC@355845525), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@49ec74c0 -> wakeup thread Time-limited test -> (continuation of Deferred@490905849 after retry RPC@933404133) -> (continuation of Deferred@450532377 after retry RPC@355845525))) after lookup
08:16:59.548 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] connecting to peer
08:16:59.549 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:59.549 [DEBUG - kudu-nio-1] (Connection.java:709) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] cleaning up while in state CONNECTING due to: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:59.550 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:59.626 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:16:59.627 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983),f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)[L],f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)]}]
08:16:59.627 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=8, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=165), Traces: [0ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [2ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [2ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [25ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [26ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [28ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [28ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [42ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [46ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [47ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [49ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [49ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [82ms] refreshing cache from master, [82ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [85ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [87ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [88ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [89ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [162ms] refreshing cache from master, [162ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [165ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@1578262747(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@4067e6b8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@490905849 after retry RPC@933404133) -> (continuation of Deferred@450532377 after retry RPC@355845525) -> (continuation of Deferred@256270869 after retry RPC@878970629), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@49ec74c0 -> wakeup thread Time-limited test -> (continuation of Deferred@490905849 after retry RPC@933404133) -> (continuation of Deferred@450532377 after retry RPC@355845525) -> (continuation of Deferred@256270869 after retry RPC@878970629))) after lookup
08:16:59.628 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] connecting to peer
08:16:59.629 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:59.629 [DEBUG - kudu-nio-2] (Connection.java:709) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] cleaning up while in state CONNECTING due to: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:16:59.629 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:17:00.066 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:17:00.068 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983),f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)[L],f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)]}]
08:17:00.068 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=10, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=606), Traces: [0ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [2ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [2ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [25ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [26ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [28ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [28ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [42ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [46ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [47ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [49ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [49ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [82ms] refreshing cache from master, [82ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [85ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [87ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [88ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [89ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [162ms] refreshing cache from master, [162ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [165ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [166ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [168ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [168ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [602ms] refreshing cache from master, [602ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [605ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@1578262747(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@4067e6b8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@490905849 after retry RPC@933404133) -> (continuation of Deferred@450532377 after retry RPC@355845525) -> (continuation of Deferred@256270869 after retry RPC@878970629) -> (continuation of Deferred@1282577404 after retry RPC@1488648536), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@49ec74c0 -> wakeup thread Time-limited test -> (continuation of Deferred@490905849 after retry RPC@933404133) -> (continuation of Deferred@450532377 after retry RPC@355845525) -> (continuation of Deferred@256270869 after retry RPC@878970629) -> (continuation of Deferred@1282577404 after retry RPC@1488648536))) after lookup
08:17:00.069 [DEBUG - kudu-nio-0] (Connection.java:748) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] connecting to peer
08:17:00.070 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:17:00.070 [DEBUG - kudu-nio-3] (Connection.java:709) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] cleaning up while in state CONNECTING due to: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:17:00.070 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247) for tablet b328961edc164a1289f259240593e73e: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247
08:17:00.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:00.652410 13507 raft_consensus.cc:493] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader f0e29bd27cb448589ac37e569ad1a164)
08:17:00.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:00.652612 13507 raft_consensus.cc:515] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:17:00.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:00.653442 13507 leader_election.cc:290] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247), f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65:39249)
08:17:00.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:17:00.654837 12869 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.40.66:46247: connect: Connection refused (error 111)
08:17:00.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:17:00.656363 12869 leader_election.cc:336] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247): Network error: Client connection negotiation failed: client connection to 127.10.40.66:46247: connect: Connection refused (error 111)
08:17:00.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:00.662062 13428 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b328961edc164a1289f259240593e73e" candidate_uuid: "2ea1996689fb4cf18b17a35790fba392" candidate_term: 12 candidate_status { last_received { term: 11 index: 40 } } ignore_live_leader: false dest_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" is_pre_election: true
08:17:00.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:00.662569 13428 raft_consensus.cc:2468] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ea1996689fb4cf18b17a35790fba392 in term 10.
08:17:00.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:00.663574 12869 leader_election.cc:304] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ea1996689fb4cf18b17a35790fba392, f2a3f45a9e3b4f2dbda7e2613df89453; no voters: f0e29bd27cb448589ac37e569ad1a164
08:17:00.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:00.663964 13507 raft_consensus.cc:2804] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 11 FOLLOWER]: Leader pre-election won for term 12
08:17:00.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:00.664109 13507 raft_consensus.cc:493] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 11 FOLLOWER]: Starting leader election (detected failure of leader f0e29bd27cb448589ac37e569ad1a164)
08:17:00.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:00.664227 13507 raft_consensus.cc:3060] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 11 FOLLOWER]: Advancing to term 12
08:17:00.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:00.667403 13507 raft_consensus.cc:515] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:17:00.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:00.668116 13507 leader_election.cc:290] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [CANDIDATE]: Term 12 election: Requested vote from peers f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247), f2a3f45a9e3b4f2dbda7e2613df89453 (127.10.40.65:39249)
08:17:00.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:00.668900 13428 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b328961edc164a1289f259240593e73e" candidate_uuid: "2ea1996689fb4cf18b17a35790fba392" candidate_term: 12 candidate_status { last_received { term: 11 index: 40 } } ignore_live_leader: false dest_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453"
08:17:00.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:00.669145 13428 raft_consensus.cc:3060] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 10 FOLLOWER]: Advancing to term 12
08:17:00.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:17:00.669696 12869 leader_election.cc:336] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247): Network error: Client connection negotiation failed: client connection to 127.10.40.66:46247: connect: Connection refused (error 111)
08:17:00.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:00.673322 13428 raft_consensus.cc:2468] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ea1996689fb4cf18b17a35790fba392 in term 12.
08:17:00.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:00.673837 12869 leader_election.cc:304] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ea1996689fb4cf18b17a35790fba392, f2a3f45a9e3b4f2dbda7e2613df89453; no voters: f0e29bd27cb448589ac37e569ad1a164
08:17:00.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:00.674209 13507 raft_consensus.cc:2804] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 12 FOLLOWER]: Leader election won for term 12
08:17:00.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:00.674750 13507 raft_consensus.cc:697] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [term 12 LEADER]: Becoming Leader. State: Replica: 2ea1996689fb4cf18b17a35790fba392, State: Running, Role: LEADER
08:17:00.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:00.675298 13507 consensus_queue.cc:237] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 11.40, Last appended by leader: 40, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:17:00.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:00.679543 10647 catalog_manager.cc:5719] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 reported cstate change: term changed from 11 to 12, leader changed from f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66) to 2ea1996689fb4cf18b17a35790fba392 (127.10.40.67). New cstate: current_term: 12 leader_uuid: "2ea1996689fb4cf18b17a35790fba392" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } health_report { overall_health: UNKNOWN } } }
08:17:01.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:01.097759 13428 raft_consensus.cc:1275] T b328961edc164a1289f259240593e73e P f2a3f45a9e3b4f2dbda7e2613df89453 [term 12 FOLLOWER]: Refusing update from remote peer 2ea1996689fb4cf18b17a35790fba392: Log matching property violated. Preceding OpId in replica: term: 10 index: 36. Preceding OpId from leader: term: 12 index: 41. (index mismatch)
08:17:01.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:01.098944 13507 consensus_queue.cc:1048] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 36, Time since last communication: 0.000s
08:17:01.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:17:01.112700 12869 consensus_peers.cc:597] T b328961edc164a1289f259240593e73e P 2ea1996689fb4cf18b17a35790fba392 -> Peer f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247): Couldn't send request to peer f0e29bd27cb448589ac37e569ad1a164. Status: Network error: Client connection negotiation failed: client connection to 127.10.40.66:46247: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
08:17:01.387 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2538) Learned about tablet b328961edc164a1289f259240593e73e for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644' with partition [<start>, <end>)
08:17:01.387 [DEBUG - kudu-nio-0] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=b328961edc164a1289f259240593e73e@[2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)[L],f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247),f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)]}]
08:17:01.388 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=b328961edc164a1289f259240593e73e, attempt=12, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=1926), Traces: [0ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [2ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [2ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [25ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [26ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [28ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [28ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [42ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [46ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [47ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [49ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [49ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [82ms] refreshing cache from master, [82ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [85ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [87ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [88ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [89ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [162ms] refreshing cache from master, [162ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [165ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [166ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [168ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [168ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [602ms] refreshing cache from master, [602ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [605ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, [607ms] sending RPC to server f0e29bd27cb448589ac37e569ad1a164, [609ms] received response from server f0e29bd27cb448589ac37e569ad1a164: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [609ms] delaying RPC due to: Network error: Failed to connect to peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247): Connection refused: /127.10.40.66:46247, [1922ms] refreshing cache from master, [1923ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.40.124:41773, [1926ms] Sub RPC GetTableLocations: received response from server master-127.10.40.124:41773: OK, deferred=Deferred@1578262747(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fea58338998@4067e6b8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@490905849 after retry RPC@933404133) -> (continuation of Deferred@450532377 after retry RPC@355845525) -> (continuation of Deferred@256270869 after retry RPC@878970629) -> (continuation of Deferred@1282577404 after retry RPC@1488648536) -> (continuation of Deferred@1178730524 after retry RPC@1237143989), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@49ec74c0 -> wakeup thread Time-limited test -> (continuation of Deferred@490905849 after retry RPC@933404133) -> (continuation of Deferred@450532377 after retry RPC@355845525) -> (continuation of Deferred@256270869 after retry RPC@878970629) -> (continuation of Deferred@1282577404 after retry RPC@1488648536) -> (continuation of Deferred@1178730524 after retry RPC@1237143989))) after lookup
08:17:01.389 [DEBUG - kudu-nio-0] (Connection.java:748) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] connecting to peer
08:17:01.390 [DEBUG - kudu-nio-4] (Connection.java:757) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] Successfully connected to peer
08:17:01.422 [DEBUG - kudu-nio-4] (Negotiator.java:918) Authenticated connection [id: 0x76144466, L:/127.0.0.1:49316 - R:/127.10.40.67:44983] using TOKEN/null
08:17:01.450 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.10.40.66:46247
08:17:01.450 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

08:17:01.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:01.451193 10401 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0lHOBl/build/asan/bin/kudu
08:17:01.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-task0lHOBl/build/asan/bin/kudu
08:17:01.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/wal
08:17:01.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/data
08:17:01.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
08:17:01.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
08:17:01.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
08:17:01.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:17:01.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:17:01.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:17:01.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/logs
08:17:01.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/data/info.pb
08:17:01.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:17:01.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
08:17:01.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
08:17:01.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
08:17:01.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
08:17:01.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:17:01.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:17:01.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:17:01.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
08:17:01.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
08:17:01.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.40.66:46247
08:17:01.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.40.66
08:17:01.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.40.126:45793,127.10.40.125:46051,127.10.40.124:41773
08:17:01.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=33605
08:17:01.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.40.66
08:17:01.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.40.84:40955
08:17:01.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:17:01.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:17:01.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
08:17:01.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:17:01.816944 13541 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
08:17:01.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:17:01.817269 13541 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
08:17:01.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:17:01.817354 13541 flags.cc:432] Enabled unsafe flag: --never_fsync=true
08:17:01.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:17:01.825872 13541 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
08:17:01.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:17:01.826030 13541 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.40.66
08:17:01.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:01.836293 13541 tablet_server_runner.cc:78] Tablet server non-default flags:
08:17:01.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/logs
08:17:01.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
08:17:01.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
08:17:01.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
08:17:01.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.40.84:40955
08:17:01.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
08:17:01.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
08:17:01.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/data
08:17:01.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/wal
08:17:01.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
08:17:01.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
08:17:01.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.40.66:46247
08:17:01.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
08:17:01.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
08:17:01.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
08:17:01.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/data/info.pb
08:17:01.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.40.66
08:17:01.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=33605
08:17:01.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.40.126:45793,127.10.40.125:46051,127.10.40.124:41773
08:17:01.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
08:17:01.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
08:17:01.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
08:17:01.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
08:17:01.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
08:17:01.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.40.66
08:17:01.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
08:17:01.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
08:17:01.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
08:17:01.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 70c51312f1186a8ae9f46647160b7730123f7601
08:17:01.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
08:17:01.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 12 Aug 2026 07:43:21 UTC on bdcb31816ec0
08:17:01.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 13302
08:17:01.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
08:17:01.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:01.838029 13541 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
08:17:01.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:01.839990 13541 file_cache.cc:504] Constructed file cache file cache with capacity 419430
08:17:01.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:17:01.854961 13546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:17:01.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:17:01.855551 13547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:17:01.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:17:01.864197 13551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
08:17:01.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:01.864722 13541 server_base.cc:1061] running on GCE node
08:17:01.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:01.865476 13541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
08:17:01.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:01.866607 13541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
08:17:01.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:01.867875 13541 hybrid_clock.cc:648] HybridClock initialized: now 1786522621867798 us; error 71 us; skew 500 ppm
08:17:01.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:01.868257 13541 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
08:17:01.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:01.871099 13541 webserver.cc:533] Webserver started at http://127.10.40.66:33605/ using document root <none> and password file <none>
08:17:01.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:01.871909 13541 fs_manager.cc:362] Metadata directory not provided
08:17:01.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:01.872053 13541 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
08:17:01.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:01.877986 13541 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.001s
08:17:01.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:01.881105 13559 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
08:17:01.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:01.882716 13541 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.002s
08:17:01.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:01.882855 13541 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/data,/tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/wal
08:17:01.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "f0e29bd27cb448589ac37e569ad1a164"
08:17:01.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-12 08:16:19 on dist-test-slave-gp6n"
08:17:01.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:01.883630 13541 fs_report.cc:389] FS layout report
08:17:01.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
08:17:01.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/wal
08:17:01.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/wal
08:17:01.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/data/data
08:17:01.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
08:17:01.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
08:17:01.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
08:17:01.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
08:17:01.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
08:17:01.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
08:17:01.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
08:17:01.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
08:17:01.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
08:17:01.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
08:17:01.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
08:17:01.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:01.908147 13541 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
08:17:01.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:01.909018 13541 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
08:17:01.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:01.909299 13541 kserver.cc:163] Server-wide thread pool size limit: 3276
08:17:01.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:01.910364 13541 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
08:17:01.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:01.912763 13567 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
08:17:01.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:01.918912 13541 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
08:17:01.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:01.919030 13541 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.007s	user 0.001s	sys 0.000s
08:17:01.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:01.919173 13541 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
08:17:01.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:01.922742 13541 ts_tablet_manager.cc:616] Registered 1 tablets
08:17:01.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:01.922868 13541 ts_tablet_manager.cc:595] Time spent register tablets: real 0.004s	user 0.004s	sys 0.000s
08:17:01.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:01.923379 13567 tablet_bootstrap.cc:492] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164: Bootstrap starting.
08:17:01.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:01.956167 13567 log.cc:826] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164: Log is configured to *not* fsync() on all Append() calls
08:17:01.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:01.994222 13541 rpc_server.cc:307] RPC server started. Bound to: 127.10.40.66:46247
08:17:01.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:01.996623 13541 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0lHOBl/test-tmp/mini-kudu-cluster1281313024778193426/ts-1/data/info.pb
08:17:01.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:01.994256 13705 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.40.66:46247 every 8 connection(s)
08:17:01.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:01.997493 10401 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0lHOBl/build/asan/bin/kudu as pid 13541
08:17:01.997 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
08:17:02.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:02.012856 13567 tablet_bootstrap.cc:492] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164: Bootstrap replayed 1/1 log segments. Stats: ops{read=40 overwritten=0 applied=40 ignored=0} inserts{seen=30 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
08:17:02.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:02.013662 13567 tablet_bootstrap.cc:492] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164: Bootstrap complete.
08:17:02.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:02.015823 13567 ts_tablet_manager.cc:1403] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164: Time spent bootstrapping tablet: real 0.093s	user 0.068s	sys 0.021s
08:17:02.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:02.024452 13567 raft_consensus.cc:359] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 11 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:17:02.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:02.025730 13567 raft_consensus.cc:740] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [term 11 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e29bd27cb448589ac37e569ad1a164, State: Initialized, Role: FOLLOWER
08:17:02.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:02.026677 13567 consensus_queue.cc:260] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 11.40, Last appended by leader: 40, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0e29bd27cb448589ac37e569ad1a164" member_type: VOTER last_known_addr { host: "127.10.40.66" port: 46247 } } peers { permanent_uuid: "2ea1996689fb4cf18b17a35790fba392" member_type: VOTER last_known_addr { host: "127.10.40.67" port: 44983 } } peers { permanent_uuid: "f2a3f45a9e3b4f2dbda7e2613df89453" member_type: VOTER last_known_addr { host: "127.10.40.65" port: 39249 } }
08:17:02.080 [DEBUG - kudu-nio-4] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644, tablet=b328961edc164a1289f259240593e73e, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:17:02.088 [DEBUG - kudu-nio-4] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786522580644, tablet=b328961edc164a1289f259240593e73e, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
08:17:02.095 [DEBUG - main] (Connection.java:575) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] disconnecting while in state READY
08:17:02.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:02.097215 13567 ts_tablet_manager.cc:1434] T b328961edc164a1289f259240593e73e P f0e29bd27cb448589ac37e569ad1a164: Time spent starting tablet: real 0.081s	user 0.022s	sys 0.027s
08:17:02.099 [DEBUG - kudu-nio-4] (Connection.java:239) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] handling channelInactive
08:17:02.101 [DEBUG - main] (Connection.java:575) [peer master-127.10.40.124:41773(127.10.40.124:41773)] disconnecting while in state READY
08:17:02.101 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 2ea1996689fb4cf18b17a35790fba392(127.10.40.67:44983)] cleaning up while in state READY due to: connection closed
08:17:02.101 [DEBUG - main] (Connection.java:575) [peer f0e29bd27cb448589ac37e569ad1a164(127.10.40.66:46247)] disconnecting while in state TERMINATED
08:17:02.101 [DEBUG - main] (Connection.java:575) [peer master-127.10.40.126:45793(127.10.40.126:45793)] disconnecting while in state READY
08:17:02.102 [DEBUG - main] (Connection.java:575) [peer f2a3f45a9e3b4f2dbda7e2613df89453(127.10.40.65:39249)] disconnecting while in state TERMINATED
08:17:02.102 [DEBUG - main] (Connection.java:575) [peer master-127.10.40.125:46051(127.10.40.125:46051)] disconnecting while in state READY
08:17:02.102 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.10.40.124:41773(127.10.40.124:41773)] handling channelInactive
08:17:02.102 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.10.40.124:41773(127.10.40.124:41773)] cleaning up while in state READY due to: connection closed
08:17:02.103 [DEBUG - kudu-nio-2] (Connection.java:239) [peer master-127.10.40.125:46051(127.10.40.125:46051)] handling channelInactive
08:17:02.103 [DEBUG - kudu-nio-2] (Connection.java:709) [peer master-127.10.40.125:46051(127.10.40.125:46051)] cleaning up while in state READY due to: connection closed
08:17:02.104 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2704) Releasing all remaining resources
08:17:02.105 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.10.40.126:45793(127.10.40.126:45793)] handling channelInactive
08:17:02.105 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.10.40.126:45793(127.10.40.126:45793)] cleaning up while in state READY due to: connection closed
08:17:02.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:02.107265 10401 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-task0lHOBl/build/asan/bin/kudu pid 13319
08:17:02.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:02.117969 13706 heartbeater.cc:344] Connected to a master server at 127.10.40.124:41773
08:17:02.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:02.118525 13706 heartbeater.cc:461] Registering TS with master...
08:17:02.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:02.119953 13718 heartbeater.cc:344] Connected to a master server at 127.10.40.125:46051
08:17:02.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:02.120142 13718 heartbeater.cc:461] Registering TS with master...
08:17:02.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:02.120786 13718 heartbeater.cc:507] Master 127.10.40.125:46051 requested a full tablet report, sending...
08:17:02.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:02.124812 13706 heartbeater.cc:507] Master 127.10.40.124:41773 requested a full tablet report, sending...
08:17:02.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:02.124955 13715 heartbeater.cc:344] Connected to a master server at 127.10.40.126:45793
08:17:02.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:02.125098 13715 heartbeater.cc:461] Registering TS with master...
08:17:02.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:02.125638 13715 heartbeater.cc:507] Master 127.10.40.126:45793 requested a full tablet report, sending...
08:17:02.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:02.127153 10452 ts_manager.cc:194] Re-registered known tserver with Master: f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247)
08:17:02.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:02.127648 10648 ts_manager.cc:194] Re-registered known tserver with Master: f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247)
08:17:02.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:02.129187 10648 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.10.40.66:49489
08:17:02.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:02.130656 10549 ts_manager.cc:194] Re-registered known tserver with Master: f0e29bd27cb448589ac37e569ad1a164 (127.10.40.66:46247)
08:17:02.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:02.132143 13706 heartbeater.cc:499] Master 127.10.40.124:41773 was elected leader, sending a full tablet report...
08:17:02.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:02.143975 13479 generic_service.cc:196] Checking for leaks (request via RPC)
08:17:02.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:02.270561 10401 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0lHOBl/build/asan/bin/kudu with pid 13319
08:17:02.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:02.306329 10401 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-task0lHOBl/build/asan/bin/kudu pid 13541
08:17:02.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:02.316416 13698 generic_service.cc:196] Checking for leaks (request via RPC)
08:17:02.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:02.410863 10401 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0lHOBl/build/asan/bin/kudu with pid 13541
08:17:02.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:02.444391 10401 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-task0lHOBl/build/asan/bin/kudu pid 12846
08:17:02.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:02.452781 13002 generic_service.cc:196] Checking for leaks (request via RPC)
08:17:02.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:02.583705 10401 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0lHOBl/build/asan/bin/kudu with pid 12846
08:17:02.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:02.612135 10401 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-task0lHOBl/build/asan/bin/kudu pid 10414
08:17:02.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:02.613440 10494 generic_service.cc:196] Checking for leaks (request via RPC)
08:17:02.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:02.738104 10401 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0lHOBl/build/asan/bin/kudu with pid 10414
08:17:02.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:02.772724 10401 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-task0lHOBl/build/asan/bin/kudu pid 10503
08:17:02.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:02.773874 10592 generic_service.cc:196] Checking for leaks (request via RPC)
08:17:02.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:17:02.843580 10630 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.40.126:45793: connect: Connection refused (error 111)
08:17:02.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260812 08:17:02.845220 10630 consensus_peers.cc:597] T 00000000000000000000000000000000 P 88955d5e7d8b4dc8a58cf089ab03559c -> Peer a0b587403b6a4d3090cfced49037efc0 (127.10.40.126:45793): Couldn't send request to peer a0b587403b6a4d3090cfced49037efc0. Status: Network error: Client connection negotiation failed: client connection to 127.10.40.126:45793: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
08:17:02.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:02.901473 10401 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0lHOBl/build/asan/bin/kudu with pid 10503
08:17:02.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:02.933972 10401 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-task0lHOBl/build/asan/bin/kudu pid 10603
08:17:02.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:02.935235 10685 generic_service.cc:196] Checking for leaks (request via RPC)
08:17:03.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260812 08:17:03.065712 10401 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0lHOBl/build/asan/bin/kudu with pid 10603
08:17:03.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-08-12T08:17:03Z chronyd exiting

Time: 48.041

OK (1 test)

