JUnit version 4.13.2
.14:00:04.722 [DEBUG - main] (RandomUtils.java:49) Using random seed: 1779112804715
14:00:04.785 [INFO - main] (RetryRule.java:138) Creating RetryStatement with result reporter and retry count of 0 (no retries)
14:00:04.844 [INFO - main] (KuduTestHarness.java:155) Creating a new MiniKuduCluster...
14:00:04.856 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskw_3vRI/test-tmp
14:00:04.862 [INFO - main] (KuduBinaryLocator.java:83) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/asan/bin
14:00:04.864 [INFO - main] (MiniKuduCluster.java:217) Starting process: [/tmp/dist-test-taskw_3vRI/java/build/dist-test/../../../build/asan/bin/kudu, test, mini_cluster, --serialization=pb]
14:00:05.092 [DEBUG - main] (MiniKuduCluster.java:185) Request: create_cluster {
  num_masters: 3
  num_tservers: 3
  enable_kerberos: false
  cluster_root: "/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694"
  hms_mode: NONE
  mini_kdc_options {
  }
  principal: "kudu"
  mini_oidc_options {
  }
}

14:00:05.300 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
14:00:05.302 [DEBUG - main] (MiniKuduCluster.java:185) Request: start_cluster {
}

14:00:05.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:05.306083 16427 mini_oidc.cc:150] Starting JWKS server
14:00:05.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:05.309424 16427 webserver.cc:533] Webserver started at http://127.0.0.1:37217/ using document root <none> and password file <none>
14:00:05.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:05.309763 16427 mini_oidc.cc:186] Starting OIDC Discovery server
14:00:05.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:05.311198 16427 webserver.cc:533] Webserver started at http://127.0.0.1:37117/ using document root <none> and password file <none>
14:00:05.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-05-18T14:00:05Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
14:00:05.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-05-18T14:00:05Z Disabled control of system clock
14:00:05.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:05.347429 16427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu
14:00:05.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu
14:00:05.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-0/wal
14:00:05.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-0/data
14:00:05.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:00:05.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:00:05.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:00:05.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:00:05.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:00:05.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:00:05.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-0/logs
14:00:05.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-0/data/info.pb
14:00:05.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:00:05.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:00:05.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:00:05.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:00:05.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:00:05.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:00:05.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:00:05.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:00:05.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
14:00:05.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:00:05.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
14:00:05.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
14:00:05.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.10.254:37885
14:00:05.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.10.254
14:00:05.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:00:05.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.10.212:33969
14:00:05.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:00:05.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:00:05.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:00:05.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
14:00:05.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.16.10.254:37885,127.16.10.253:33517,127.16.10.252:36395
14:00:05.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
14:00:05.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:37217/jwks.json with env {}
14:00:05.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:05.689463 16438 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:00:05.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:05.689810 16438 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:00:05.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:05.689885 16438 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:00:05.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:05.699257 16438 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
14:00:05.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:05.699362 16438 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:00:05.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:05.699417 16438 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
14:00:05.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:05.699458 16438 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
14:00:05.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:05.699509 16438 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
14:00:05.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:05.699542 16438 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:37217/jwks.json
14:00:05.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:05.710561 16438 master_runner.cc:387] Master server non-default flags:
14:00:05.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:00:05.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.10.212:33969
14:00:05.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:00:05.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:00:05.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-0/data
14:00:05.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-0/wal
14:00:05.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
14:00:05.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.16.10.254:37885,127.16.10.253:33517,127.16.10.252:36395
14:00:05.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:00:05.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:00:05.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
14:00:05.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.10.254:37885
14:00:05.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
14:00:05.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:00:05.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
14:00:05.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:37217/jwks.json
14:00:05.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:00:05.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:00:05.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-0/data/info.pb
14:00:05.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.10.254
14:00:05.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:00:05.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:00:05.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:00:05.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:00:05.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:00:05.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:00:05.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-0/logs
14:00:05.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:00:05.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:00:05.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:00:05.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
14:00:05.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:00:05.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
14:00:05.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:00:05.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 13:43:23 UTC on bdcb31816ec0
14:00:05.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11958
14:00:05.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
14:00:05.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:05.712584 16438 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:00:05.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:05.715225 16438 file_cache.cc:492] Constructed file cache file cache with capacity 419430
14:00:05.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:05.725994 16444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:00:05.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:05.726024 16446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:00:05.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:05.726102 16443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:00:05.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:05.727890 16438 server_base.cc:1061] running on GCE node
14:00:05.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:05.728917 16438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:00:05.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:05.730567 16438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:00:05.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:05.731833 16438 hybrid_clock.cc:648] HybridClock initialized: now 1779112805731734 us; error 93 us; skew 500 ppm
14:00:05.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:05.732184 16438 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:00:05.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:05.734606 16438 webserver.cc:533] Webserver started at http://127.16.10.254:35859/ using document root <none> and password file <none>
14:00:05.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:05.735337 16438 fs_manager.cc:362] Metadata directory not provided
14:00:05.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:05.735441 16438 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:00:05.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:05.735843 16438 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
14:00:05.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:05.738231 16438 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-0/data/instance:
14:00:05.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "5f66fba063cb4d5599b228308545c515"
14:00:05.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:00:05 on dist-test-slave-rq0s"
14:00:05.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:05.739236 16438 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-0/wal/instance:
14:00:05.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "5f66fba063cb4d5599b228308545c515"
14:00:05.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:00:05 on dist-test-slave-rq0s"
14:00:05.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:05.745044 16438 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
14:00:05.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:05.748749 16452 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:00:05.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:05.750633 16438 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
14:00:05.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:05.750797 16438 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-0/data,/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-0/wal
14:00:05.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "5f66fba063cb4d5599b228308545c515"
14:00:05.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:00:05 on dist-test-slave-rq0s"
14:00:05.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:05.750970 16438 fs_report.cc:389] FS layout report
14:00:05.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:00:05.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-0/wal
14:00:05.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-0/wal
14:00:05.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-0/data/data
14:00:05.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:00:05.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:00:05.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:00:05.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:00:05.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:00:05.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:00:05.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:00:05.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:00:05.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:00:05.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:00:05.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:00:05.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:05.797029 16438 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:00:05.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:05.797982 16438 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:00:05.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:05.798233 16438 kserver.cc:163] Server-wide thread pool size limit: 3276
14:00:05.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:05.819504 16438 rpc_server.cc:307] RPC server started. Bound to: 127.16.10.254:37885
14:00:05.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:05.819526 16503 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.10.254:37885 every 8 connection(s)
14:00:05.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:05.821386 16438 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-0/data/info.pb
14:00:05.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:05.826071 16504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
14:00:05.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:05.828748 16427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu as pid 16438
14:00:05.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:05.829110 16427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-0/wal/instance
14:00:05.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:05.831880 16427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu
14:00:05.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu
14:00:05.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-1/wal
14:00:05.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-1/data
14:00:05.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:00:05.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:00:05.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:00:05.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:00:05.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:00:05.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:00:05.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-1/logs
14:00:05.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-1/data/info.pb
14:00:05.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:00:05.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:00:05.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:00:05.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:00:05.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:00:05.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:00:05.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:00:05.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:00:05.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
14:00:05.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:00:05.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
14:00:05.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
14:00:05.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.10.253:33517
14:00:05.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.10.253
14:00:05.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:00:05.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.10.212:33969
14:00:05.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:00:05.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:00:05.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:00:05.839 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
14:00:05.839 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.16.10.254:37885,127.16.10.253:33517,127.16.10.252:36395
14:00:05.839 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
14:00:05.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:37217/jwks.json with env {}
14:00:05.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:05.832028 16504 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.10.254" port: 37885 } has no permanent_uuid. Determining permanent_uuid...
14:00:05.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:05.855835 16504 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.10.253" port: 33517 } has no permanent_uuid. Determining permanent_uuid...
14:00:05.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:05.857227 16455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.10.253:33517: connect: Connection refused (error 111)
14:00:05.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:05.859351 16504 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.16.10.253:33517: Network error: Client connection negotiation failed: client connection to 127.16.10.253:33517: connect: Connection refused (error 111)
14:00:05.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:05.908789 16504 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.10.253" port: 33517 } attempt: 1
14:00:05.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:05.911108 16504 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.16.10.253:33517: Network error: Client connection negotiation failed: client connection to 127.16.10.253:33517: connect: Connection refused (error 111)
14:00:05.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:05.979403 16504 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.10.253" port: 33517 } attempt: 2
14:00:05.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:05.982273 16504 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.16.10.253:33517: Network error: Client connection negotiation failed: client connection to 127.16.10.253:33517: connect: Connection refused (error 111)
14:00:06.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.073621 16504 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.10.253" port: 33517 } attempt: 3
14:00:06.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:06.075953 16504 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.16.10.253:33517: Network error: Client connection negotiation failed: client connection to 127.16.10.253:33517: connect: Connection refused (error 111)
14:00:06.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:06.178049 16507 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:00:06.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:06.178414 16507 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:00:06.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:06.178522 16507 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:00:06.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:06.187358 16507 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
14:00:06.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:06.187539 16507 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:00:06.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:06.187625 16507 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
14:00:06.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:06.187686 16507 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
14:00:06.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:06.187767 16507 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
14:00:06.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:06.187816 16507 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:37217/jwks.json
14:00:06.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.198794 16507 master_runner.cc:387] Master server non-default flags:
14:00:06.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:00:06.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.10.212:33969
14:00:06.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:00:06.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:00:06.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-1/data
14:00:06.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-1/wal
14:00:06.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
14:00:06.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.16.10.254:37885,127.16.10.253:33517,127.16.10.252:36395
14:00:06.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:00:06.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:00:06.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
14:00:06.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.10.253:33517
14:00:06.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
14:00:06.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:00:06.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
14:00:06.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:37217/jwks.json
14:00:06.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:00:06.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:00:06.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-1/data/info.pb
14:00:06.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.10.253
14:00:06.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:00:06.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:00:06.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:00:06.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:00:06.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:00:06.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:00:06.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-1/logs
14:00:06.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:00:06.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:00:06.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:00:06.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
14:00:06.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:00:06.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
14:00:06.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:00:06.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 13:43:23 UTC on bdcb31816ec0
14:00:06.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11958
14:00:06.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
14:00:06.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.201200 16507 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:00:06.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.203939 16507 file_cache.cc:492] Constructed file cache file cache with capacity 419430
14:00:06.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:06.213438 16514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:00:06.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:06.215212 16515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:00:06.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:06.217334 16517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:00:06.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.218524 16507 server_base.cc:1061] running on GCE node
14:00:06.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.219216 16504 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.10.253" port: 33517 } attempt: 4
14:00:06.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.219599 16507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:00:06.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:06.221366 16504 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.16.10.253:33517: Network error: Client connection negotiation failed: client connection to 127.16.10.253:33517: connect: Connection refused (error 111)
14:00:06.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.221755 16507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:00:06.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.222956 16507 hybrid_clock.cc:648] HybridClock initialized: now 1779112806222895 us; error 51 us; skew 500 ppm
14:00:06.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.223273 16507 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:00:06.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.225792 16507 webserver.cc:533] Webserver started at http://127.16.10.253:43955/ using document root <none> and password file <none>
14:00:06.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.226676 16507 fs_manager.cc:362] Metadata directory not provided
14:00:06.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.226775 16507 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:00:06.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.227167 16507 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
14:00:06.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.230638 16507 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-1/data/instance:
14:00:06.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "ffb833f1426c4a6ea62563ed7654c0f2"
14:00:06.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:00:06 on dist-test-slave-rq0s"
14:00:06.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.231805 16507 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-1/wal/instance:
14:00:06.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "ffb833f1426c4a6ea62563ed7654c0f2"
14:00:06.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:00:06 on dist-test-slave-rq0s"
14:00:06.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.238602 16507 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.006s	sys 0.001s
14:00:06.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.242440 16523 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:00:06.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.244073 16507 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.003s
14:00:06.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.244221 16507 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-1/data,/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-1/wal
14:00:06.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "ffb833f1426c4a6ea62563ed7654c0f2"
14:00:06.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:00:06 on dist-test-slave-rq0s"
14:00:06.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.244383 16507 fs_report.cc:389] FS layout report
14:00:06.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:00:06.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-1/wal
14:00:06.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-1/wal
14:00:06.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-1/data/data
14:00:06.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:00:06.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:00:06.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:00:06.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:00:06.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:00:06.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:00:06.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:00:06.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:00:06.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:00:06.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:00:06.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:00:06.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.264539 16507 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:00:06.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.265491 16507 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:00:06.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.265784 16507 kserver.cc:163] Server-wide thread pool size limit: 3276
14:00:06.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.286231 16507 rpc_server.cc:307] RPC server started. Bound to: 127.16.10.253:33517
14:00:06.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.286274 16574 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.10.253:33517 every 8 connection(s)
14:00:06.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.288062 16507 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-1/data/info.pb
14:00:06.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.292743 16575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
14:00:06.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.294004 16427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu as pid 16507
14:00:06.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.294216 16427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-1/wal/instance
14:00:06.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.295352 16427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu
14:00:06.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu
14:00:06.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-2/wal
14:00:06.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-2/data
14:00:06.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:00:06.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:00:06.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:00:06.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:00:06.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:00:06.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:00:06.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-2/logs
14:00:06.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-2/data/info.pb
14:00:06.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:00:06.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:00:06.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:00:06.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:00:06.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:00:06.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:00:06.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:00:06.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:00:06.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
14:00:06.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:00:06.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
14:00:06.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
14:00:06.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.10.252:36395
14:00:06.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.10.252
14:00:06.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:00:06.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.10.212:33969
14:00:06.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:00:06.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:00:06.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:00:06.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
14:00:06.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.16.10.254:37885,127.16.10.253:33517,127.16.10.252:36395
14:00:06.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
14:00:06.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:37217/jwks.json with env {}
14:00:06.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.300081 16575 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.10.254" port: 37885 } has no permanent_uuid. Determining permanent_uuid...
14:00:06.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.318503 16575 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.10.253" port: 33517 } has no permanent_uuid. Determining permanent_uuid...
14:00:06.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.329152 16575 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.10.252" port: 36395 } has no permanent_uuid. Determining permanent_uuid...
14:00:06.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:06.330224 16527 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.10.252:36395: connect: Connection refused (error 111)
14:00:06.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:06.331879 16575 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.16.10.252:36395: Network error: Client connection negotiation failed: client connection to 127.16.10.252:36395: connect: Connection refused (error 111)
14:00:06.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.381263 16575 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.10.252" port: 36395 } attempt: 1
14:00:06.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:06.383606 16575 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.16.10.252:36395: Network error: Client connection negotiation failed: client connection to 127.16.10.252:36395: connect: Connection refused (error 111)
14:00:06.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.451901 16575 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.10.252" port: 36395 } attempt: 2
14:00:06.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:06.456094 16575 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.16.10.252:36395: Network error: Client connection negotiation failed: client connection to 127.16.10.252:36395: connect: Connection refused (error 111)
14:00:06.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.520671 16504 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.10.253" port: 33517 } attempt: 5
14:00:06.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.527855 16504 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.10.252" port: 36395 } has no permanent_uuid. Determining permanent_uuid...
14:00:06.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:06.529836 16504 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.16.10.252:36395: Network error: Client connection negotiation failed: client connection to 127.16.10.252:36395: connect: Connection refused (error 111)
14:00:06.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.547375 16575 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.10.252" port: 36395 } attempt: 3
14:00:06.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:06.549844 16575 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.16.10.252:36395: Network error: Client connection negotiation failed: client connection to 127.16.10.252:36395: connect: Connection refused (error 111)
14:00:06.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.581138 16504 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.10.252" port: 36395 } attempt: 1
14:00:06.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:06.583493 16504 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.16.10.252:36395: Network error: Client connection negotiation failed: client connection to 127.16.10.252:36395: connect: Connection refused (error 111)
14:00:06.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:06.648396 16577 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:00:06.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:06.648751 16577 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:00:06.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:06.648830 16577 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:00:06.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.651768 16504 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.10.252" port: 36395 } attempt: 2
14:00:06.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:06.653863 16504 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.16.10.252:36395: Network error: Client connection negotiation failed: client connection to 127.16.10.252:36395: connect: Connection refused (error 111)
14:00:06.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:06.658285 16577 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
14:00:06.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:06.658390 16577 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:00:06.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:06.658447 16577 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
14:00:06.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:06.658561 16577 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
14:00:06.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:06.658643 16577 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
14:00:06.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:06.658694 16577 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:37217/jwks.json
14:00:06.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.670279 16577 master_runner.cc:387] Master server non-default flags:
14:00:06.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:00:06.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.10.212:33969
14:00:06.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:00:06.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:00:06.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-2/data
14:00:06.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-2/wal
14:00:06.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
14:00:06.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.16.10.254:37885,127.16.10.253:33517,127.16.10.252:36395
14:00:06.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:00:06.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:00:06.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
14:00:06.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.10.252:36395
14:00:06.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
14:00:06.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:00:06.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
14:00:06.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:37217/jwks.json
14:00:06.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:00:06.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:00:06.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-2/data/info.pb
14:00:06.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.10.252
14:00:06.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:00:06.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:00:06.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:00:06.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:00:06.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:00:06.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:00:06.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-2/logs
14:00:06.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:00:06.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:00:06.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:00:06.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
14:00:06.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:00:06.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
14:00:06.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:00:06.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 13:43:23 UTC on bdcb31816ec0
14:00:06.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11958
14:00:06.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
14:00:06.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.672420 16577 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:00:06.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.675026 16577 file_cache.cc:492] Constructed file cache file cache with capacity 419430
14:00:06.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:06.684654 16586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:00:06.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:06.684715 16585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:00:06.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:06.684831 16588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:00:06.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.685670 16577 server_base.cc:1061] running on GCE node
14:00:06.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.686453 16577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:00:06.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.688064 16577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:00:06.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.689278 16577 hybrid_clock.cc:648] HybridClock initialized: now 1779112806689217 us; error 62 us; skew 500 ppm
14:00:06.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.689625 16577 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:00:06.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.692186 16577 webserver.cc:533] Webserver started at http://127.16.10.252:43585/ using document root <none> and password file <none>
14:00:06.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.692946 16577 fs_manager.cc:362] Metadata directory not provided
14:00:06.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.693053 16577 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:00:06.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.693130 16575 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.10.252" port: 36395 } attempt: 4
14:00:06.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.693365 16577 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
14:00:06.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:06.695439 16575 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.16.10.252:36395: Network error: Client connection negotiation failed: client connection to 127.16.10.252:36395: connect: Connection refused (error 111)
14:00:06.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.696468 16577 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-2/data/instance:
14:00:06.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b98590844ccb4749a2a11ebcc6cd60cb"
14:00:06.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:00:06 on dist-test-slave-rq0s"
14:00:06.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.697294 16577 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-2/wal/instance:
14:00:06.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b98590844ccb4749a2a11ebcc6cd60cb"
14:00:06.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:00:06 on dist-test-slave-rq0s"
14:00:06.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.703001 16577 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.001s	sys 0.006s
14:00:06.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.706615 16594 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:00:06.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.708243 16577 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
14:00:06.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.708408 16577 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-2/data,/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-2/wal
14:00:06.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b98590844ccb4749a2a11ebcc6cd60cb"
14:00:06.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:00:06 on dist-test-slave-rq0s"
14:00:06.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.708587 16577 fs_report.cc:389] FS layout report
14:00:06.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:00:06.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-2/wal
14:00:06.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-2/wal
14:00:06.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-2/data/data
14:00:06.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:00:06.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:00:06.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:00:06.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:00:06.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:00:06.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:00:06.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:00:06.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:00:06.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:00:06.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:00:06.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:00:06.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.734099 16577 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:00:06.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.735349 16577 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:00:06.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.735606 16577 kserver.cc:163] Server-wide thread pool size limit: 3276
14:00:06.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.758435 16645 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.10.252:36395 every 8 connection(s)
14:00:06.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.758504 16577 rpc_server.cc:307] RPC server started. Bound to: 127.16.10.252:36395
14:00:06.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.760118 16504 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.10.252" port: 36395 } attempt: 3
14:00:06.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.760488 16577 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-2/data/info.pb
14:00:06.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.765607 16646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
14:00:06.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.766170 16427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu as pid 16577
14:00:06.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.766382 16427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/master-2/wal/instance
14:00:06.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.773927 16646 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.10.254" port: 37885 } has no permanent_uuid. Determining permanent_uuid...
14:00:06.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.785174 16504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f66fba063cb4d5599b228308545c515: Bootstrap starting.
14:00:06.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.787539 16646 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.10.253" port: 33517 } has no permanent_uuid. Determining permanent_uuid...
14:00:06.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.790112 16504 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f66fba063cb4d5599b228308545c515: Neither blocks nor log segments found. Creating new log.
14:00:06.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.792317 16504 log.cc:826] T 00000000000000000000000000000000 P 5f66fba063cb4d5599b228308545c515: Log is configured to *not* fsync() on all Append() calls
14:00:06.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.795985 16504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f66fba063cb4d5599b228308545c515: No bootstrap required, opened a new log
14:00:06.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.796772 16646 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.10.252" port: 36395 } has no permanent_uuid. Determining permanent_uuid...
14:00:06.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.804396 16504 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5f66fba063cb4d5599b228308545c515 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f66fba063cb4d5599b228308545c515" member_type: VOTER last_known_addr { host: "127.16.10.254" port: 37885 } } peers { permanent_uuid: "ffb833f1426c4a6ea62563ed7654c0f2" member_type: VOTER last_known_addr { host: "127.16.10.253" port: 33517 } } peers { permanent_uuid: "b98590844ccb4749a2a11ebcc6cd60cb" member_type: VOTER last_known_addr { host: "127.16.10.252" port: 36395 } }
14:00:06.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.804863 16504 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5f66fba063cb4d5599b228308545c515 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
14:00:06.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.804993 16504 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5f66fba063cb4d5599b228308545c515 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f66fba063cb4d5599b228308545c515, State: Initialized, Role: FOLLOWER
14:00:06.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.806022 16504 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f66fba063cb4d5599b228308545c515 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f66fba063cb4d5599b228308545c515" member_type: VOTER last_known_addr { host: "127.16.10.254" port: 37885 } } peers { permanent_uuid: "ffb833f1426c4a6ea62563ed7654c0f2" member_type: VOTER last_known_addr { host: "127.16.10.253" port: 33517 } } peers { permanent_uuid: "b98590844ccb4749a2a11ebcc6cd60cb" member_type: VOTER last_known_addr { host: "127.16.10.252" port: 36395 } }
14:00:06.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.807210 16655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f66fba063cb4d5599b228308545c515 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f66fba063cb4d5599b228308545c515" member_type: VOTER last_known_addr { host: "127.16.10.254" port: 37885 } } peers { permanent_uuid: "ffb833f1426c4a6ea62563ed7654c0f2" member_type: VOTER last_known_addr { host: "127.16.10.253" port: 33517 } } peers { permanent_uuid: "b98590844ccb4749a2a11ebcc6cd60cb" member_type: VOTER last_known_addr { host: "127.16.10.252" port: 36395 } } }
14:00:06.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.807789 16655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f66fba063cb4d5599b228308545c515 [sys.catalog]: This master's current role is: FOLLOWER
14:00:06.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.808782 16504 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5f66fba063cb4d5599b228308545c515 [sys.catalog]: configured and running, proceeding with master startup.
14:00:06.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.813197 16646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b98590844ccb4749a2a11ebcc6cd60cb: Bootstrap starting.
14:00:06.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.819082 16646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b98590844ccb4749a2a11ebcc6cd60cb: Neither blocks nor log segments found. Creating new log.
14:00:06.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.821239 16646 log.cc:826] T 00000000000000000000000000000000 P b98590844ccb4749a2a11ebcc6cd60cb: Log is configured to *not* fsync() on all Append() calls
14:00:06.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.831204 16646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b98590844ccb4749a2a11ebcc6cd60cb: No bootstrap required, opened a new log
14:00:06.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.829066 16438 master_runner.cc:187] Error getting master registration for 127.16.10.253:33517: OK, instance_id { permanent_uuid: "ffb833f1426c4a6ea62563ed7654c0f2" instance_seqno: 1779112806280275 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
14:00:06.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.837644 16438 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
14:00:06.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.839872 16646 raft_consensus.cc:359] T 00000000000000000000000000000000 P b98590844ccb4749a2a11ebcc6cd60cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f66fba063cb4d5599b228308545c515" member_type: VOTER last_known_addr { host: "127.16.10.254" port: 37885 } } peers { permanent_uuid: "ffb833f1426c4a6ea62563ed7654c0f2" member_type: VOTER last_known_addr { host: "127.16.10.253" port: 33517 } } peers { permanent_uuid: "b98590844ccb4749a2a11ebcc6cd60cb" member_type: VOTER last_known_addr { host: "127.16.10.252" port: 36395 } }
14:00:06.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.840286 16646 raft_consensus.cc:385] T 00000000000000000000000000000000 P b98590844ccb4749a2a11ebcc6cd60cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
14:00:06.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.840427 16646 raft_consensus.cc:740] T 00000000000000000000000000000000 P b98590844ccb4749a2a11ebcc6cd60cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b98590844ccb4749a2a11ebcc6cd60cb, State: Initialized, Role: FOLLOWER
14:00:06.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.841787 16646 consensus_queue.cc:260] T 00000000000000000000000000000000 P b98590844ccb4749a2a11ebcc6cd60cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f66fba063cb4d5599b228308545c515" member_type: VOTER last_known_addr { host: "127.16.10.254" port: 37885 } } peers { permanent_uuid: "ffb833f1426c4a6ea62563ed7654c0f2" member_type: VOTER last_known_addr { host: "127.16.10.253" port: 33517 } } peers { permanent_uuid: "b98590844ccb4749a2a11ebcc6cd60cb" member_type: VOTER last_known_addr { host: "127.16.10.252" port: 36395 } }
14:00:06.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.846414 16646 sys_catalog.cc:565] T 00000000000000000000000000000000 P b98590844ccb4749a2a11ebcc6cd60cb [sys.catalog]: configured and running, proceeding with master startup.
14:00:06.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.846349 16669 sys_catalog.cc:455] T 00000000000000000000000000000000 P b98590844ccb4749a2a11ebcc6cd60cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f66fba063cb4d5599b228308545c515" member_type: VOTER last_known_addr { host: "127.16.10.254" port: 37885 } } peers { permanent_uuid: "ffb833f1426c4a6ea62563ed7654c0f2" member_type: VOTER last_known_addr { host: "127.16.10.253" port: 33517 } } peers { permanent_uuid: "b98590844ccb4749a2a11ebcc6cd60cb" member_type: VOTER last_known_addr { host: "127.16.10.252" port: 36395 } } }
14:00:06.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:06.848887 16666 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 5f66fba063cb4d5599b228308545c515: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
14:00:06.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:06.849014 16666 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
14:00:06.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.849169 16669 sys_catalog.cc:458] T 00000000000000000000000000000000 P b98590844ccb4749a2a11ebcc6cd60cb [sys.catalog]: This master's current role is: FOLLOWER
14:00:06.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:06.859494 16680 catalog_manager.cc:1593] T 00000000000000000000000000000000 P b98590844ccb4749a2a11ebcc6cd60cb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
14:00:06.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:06.860417 16680 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
14:00:06.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.861081 16577 master_runner.cc:187] Error getting master registration for 127.16.10.253:33517: OK, instance_id { permanent_uuid: "ffb833f1426c4a6ea62563ed7654c0f2" instance_seqno: 1779112806280275 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
14:00:06.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.868705 16577 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
14:00:06.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.892061 16655 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5f66fba063cb4d5599b228308545c515 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
14:00:06.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.892409 16655 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5f66fba063cb4d5599b228308545c515 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f66fba063cb4d5599b228308545c515" member_type: VOTER last_known_addr { host: "127.16.10.254" port: 37885 } } peers { permanent_uuid: "ffb833f1426c4a6ea62563ed7654c0f2" member_type: VOTER last_known_addr { host: "127.16.10.253" port: 33517 } } peers { permanent_uuid: "b98590844ccb4749a2a11ebcc6cd60cb" member_type: VOTER last_known_addr { host: "127.16.10.252" port: 36395 } }
14:00:06.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.894490 16655 leader_election.cc:290] T 00000000000000000000000000000000 P 5f66fba063cb4d5599b228308545c515 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffb833f1426c4a6ea62563ed7654c0f2 (127.16.10.253:33517), b98590844ccb4749a2a11ebcc6cd60cb (127.16.10.252:36395)
14:00:06.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.895643 16621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5f66fba063cb4d5599b228308545c515" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b98590844ccb4749a2a11ebcc6cd60cb" is_pre_election: true
14:00:06.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.896301 16621 raft_consensus.cc:2468] T 00000000000000000000000000000000 P b98590844ccb4749a2a11ebcc6cd60cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f66fba063cb4d5599b228308545c515 in term 0.
14:00:06.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.897112 16456 leader_election.cc:304] T 00000000000000000000000000000000 P 5f66fba063cb4d5599b228308545c515 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f66fba063cb4d5599b228308545c515, b98590844ccb4749a2a11ebcc6cd60cb; no voters: 
14:00:06.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.897627 16655 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5f66fba063cb4d5599b228308545c515 [term 0 FOLLOWER]: Leader pre-election won for term 1
14:00:06.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.897795 16655 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5f66fba063cb4d5599b228308545c515 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
14:00:06.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.897899 16655 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5f66fba063cb4d5599b228308545c515 [term 0 FOLLOWER]: Advancing to term 1
14:00:06.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.901317 16655 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5f66fba063cb4d5599b228308545c515 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f66fba063cb4d5599b228308545c515" member_type: VOTER last_known_addr { host: "127.16.10.254" port: 37885 } } peers { permanent_uuid: "ffb833f1426c4a6ea62563ed7654c0f2" member_type: VOTER last_known_addr { host: "127.16.10.253" port: 33517 } } peers { permanent_uuid: "b98590844ccb4749a2a11ebcc6cd60cb" member_type: VOTER last_known_addr { host: "127.16.10.252" port: 36395 } }
14:00:06.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.902244 16655 leader_election.cc:290] T 00000000000000000000000000000000 P 5f66fba063cb4d5599b228308545c515 [CANDIDATE]: Term 1 election: Requested vote from peers ffb833f1426c4a6ea62563ed7654c0f2 (127.16.10.253:33517), b98590844ccb4749a2a11ebcc6cd60cb (127.16.10.252:36395)
14:00:06.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.904709 16621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5f66fba063cb4d5599b228308545c515" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b98590844ccb4749a2a11ebcc6cd60cb"
14:00:06.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.904945 16621 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b98590844ccb4749a2a11ebcc6cd60cb [term 0 FOLLOWER]: Advancing to term 1
14:00:06.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.908443 16621 raft_consensus.cc:2468] T 00000000000000000000000000000000 P b98590844ccb4749a2a11ebcc6cd60cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f66fba063cb4d5599b228308545c515 in term 1.
14:00:06.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.909070 16456 leader_election.cc:304] T 00000000000000000000000000000000 P 5f66fba063cb4d5599b228308545c515 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f66fba063cb4d5599b228308545c515, b98590844ccb4749a2a11ebcc6cd60cb; no voters: 
14:00:06.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.910079 16655 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5f66fba063cb4d5599b228308545c515 [term 1 FOLLOWER]: Leader election won for term 1
14:00:06.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.916365 16655 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5f66fba063cb4d5599b228308545c515 [term 1 LEADER]: Becoming Leader. State: Replica: 5f66fba063cb4d5599b228308545c515, State: Running, Role: LEADER
14:00:06.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.917594 16655 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f66fba063cb4d5599b228308545c515 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f66fba063cb4d5599b228308545c515" member_type: VOTER last_known_addr { host: "127.16.10.254" port: 37885 } } peers { permanent_uuid: "ffb833f1426c4a6ea62563ed7654c0f2" member_type: VOTER last_known_addr { host: "127.16.10.253" port: 33517 } } peers { permanent_uuid: "b98590844ccb4749a2a11ebcc6cd60cb" member_type: VOTER last_known_addr { host: "127.16.10.252" port: 36395 } }
14:00:06.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.925194 16682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f66fba063cb4d5599b228308545c515 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f66fba063cb4d5599b228308545c515. Latest consensus state: current_term: 1 leader_uuid: "5f66fba063cb4d5599b228308545c515" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f66fba063cb4d5599b228308545c515" member_type: VOTER last_known_addr { host: "127.16.10.254" port: 37885 } } peers { permanent_uuid: "ffb833f1426c4a6ea62563ed7654c0f2" member_type: VOTER last_known_addr { host: "127.16.10.253" port: 33517 } } peers { permanent_uuid: "b98590844ccb4749a2a11ebcc6cd60cb" member_type: VOTER last_known_addr { host: "127.16.10.252" port: 36395 } } }
14:00:06.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.925619 16682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f66fba063cb4d5599b228308545c515 [sys.catalog]: This master's current role is: LEADER
14:00:06.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.935846 16684 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
14:00:06.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.940178 16684 catalog_manager.cc:1519] Initializing Kudu cluster ID...
14:00:06.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.953169 16621 raft_consensus.cc:1275] T 00000000000000000000000000000000 P b98590844ccb4749a2a11ebcc6cd60cb [term 1 FOLLOWER]: Refusing update from remote peer 5f66fba063cb4d5599b228308545c515: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
14:00:06.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.954420 16682 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 5f66fba063cb4d5599b228308545c515 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b98590844ccb4749a2a11ebcc6cd60cb" member_type: VOTER last_known_addr { host: "127.16.10.252" port: 36395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
14:00:06.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.977381 16669 sys_catalog.cc:455] T 00000000000000000000000000000000 P b98590844ccb4749a2a11ebcc6cd60cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f66fba063cb4d5599b228308545c515. Latest consensus state: current_term: 1 leader_uuid: "5f66fba063cb4d5599b228308545c515" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f66fba063cb4d5599b228308545c515" member_type: VOTER last_known_addr { host: "127.16.10.254" port: 37885 } } peers { permanent_uuid: "ffb833f1426c4a6ea62563ed7654c0f2" member_type: VOTER last_known_addr { host: "127.16.10.253" port: 33517 } } peers { permanent_uuid: "b98590844ccb4749a2a11ebcc6cd60cb" member_type: VOTER last_known_addr { host: "127.16.10.252" port: 36395 } } }
14:00:06.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.977670 16669 sys_catalog.cc:458] T 00000000000000000000000000000000 P b98590844ccb4749a2a11ebcc6cd60cb [sys.catalog]: This master's current role is: FOLLOWER
14:00:06.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.980183 16685 mvcc.cc:204] Tried to move back new op lower bound from 7287246057268482048 to 7287246057156583424. Current Snapshot: MvccSnapshot[applied={T|T < 7287246057268482048}]
14:00:06.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.994851 16575 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.10.252" port: 36395 } attempt: 5
14:00:06.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.995223 16669 sys_catalog.cc:455] T 00000000000000000000000000000000 P b98590844ccb4749a2a11ebcc6cd60cb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5f66fba063cb4d5599b228308545c515" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f66fba063cb4d5599b228308545c515" member_type: VOTER last_known_addr { host: "127.16.10.254" port: 37885 } } peers { permanent_uuid: "ffb833f1426c4a6ea62563ed7654c0f2" member_type: VOTER last_known_addr { host: "127.16.10.253" port: 33517 } } peers { permanent_uuid: "b98590844ccb4749a2a11ebcc6cd60cb" member_type: VOTER last_known_addr { host: "127.16.10.252" port: 36395 } } }
14:00:06.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:06.995455 16669 sys_catalog.cc:458] T 00000000000000000000000000000000 P b98590844ccb4749a2a11ebcc6cd60cb [sys.catalog]: This master's current role is: FOLLOWER
14:00:07.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.005713 16682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f66fba063cb4d5599b228308545c515 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5f66fba063cb4d5599b228308545c515" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f66fba063cb4d5599b228308545c515" member_type: VOTER last_known_addr { host: "127.16.10.254" port: 37885 } } peers { permanent_uuid: "ffb833f1426c4a6ea62563ed7654c0f2" member_type: VOTER last_known_addr { host: "127.16.10.253" port: 33517 } } peers { permanent_uuid: "b98590844ccb4749a2a11ebcc6cd60cb" member_type: VOTER last_known_addr { host: "127.16.10.252" port: 36395 } } }
14:00:07.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.006008 16682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f66fba063cb4d5599b228308545c515 [sys.catalog]: This master's current role is: LEADER
14:00:07.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.011694 16575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffb833f1426c4a6ea62563ed7654c0f2: Bootstrap starting.
14:00:07.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.012403 16684 catalog_manager.cc:1382] Generated new cluster ID: 0328278874934e8bbd585811d9ee778d
14:00:07.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.012524 16684 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
14:00:07.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.022598 16575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffb833f1426c4a6ea62563ed7654c0f2: Neither blocks nor log segments found. Creating new log.
14:00:07.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.026875 16575 log.cc:826] T 00000000000000000000000000000000 P ffb833f1426c4a6ea62563ed7654c0f2: Log is configured to *not* fsync() on all Append() calls
14:00:07.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.037377 16684 catalog_manager.cc:1405] Generated new certificate authority record
14:00:07.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.042673 16684 catalog_manager.cc:1539] Loading token signing keys...
14:00:07.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.055214 16575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffb833f1426c4a6ea62563ed7654c0f2: No bootstrap required, opened a new log
14:00:07.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.068707 16575 raft_consensus.cc:359] T 00000000000000000000000000000000 P ffb833f1426c4a6ea62563ed7654c0f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f66fba063cb4d5599b228308545c515" member_type: VOTER last_known_addr { host: "127.16.10.254" port: 37885 } } peers { permanent_uuid: "ffb833f1426c4a6ea62563ed7654c0f2" member_type: VOTER last_known_addr { host: "127.16.10.253" port: 33517 } } peers { permanent_uuid: "b98590844ccb4749a2a11ebcc6cd60cb" member_type: VOTER last_known_addr { host: "127.16.10.252" port: 36395 } }
14:00:07.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.070052 16575 raft_consensus.cc:385] T 00000000000000000000000000000000 P ffb833f1426c4a6ea62563ed7654c0f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
14:00:07.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.070675 16575 raft_consensus.cc:740] T 00000000000000000000000000000000 P ffb833f1426c4a6ea62563ed7654c0f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffb833f1426c4a6ea62563ed7654c0f2, State: Initialized, Role: FOLLOWER
14:00:07.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.077106 16575 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffb833f1426c4a6ea62563ed7654c0f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f66fba063cb4d5599b228308545c515" member_type: VOTER last_known_addr { host: "127.16.10.254" port: 37885 } } peers { permanent_uuid: "ffb833f1426c4a6ea62563ed7654c0f2" member_type: VOTER last_known_addr { host: "127.16.10.253" port: 33517 } } peers { permanent_uuid: "b98590844ccb4749a2a11ebcc6cd60cb" member_type: VOTER last_known_addr { host: "127.16.10.252" port: 36395 } }
14:00:07.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.091240 16691 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffb833f1426c4a6ea62563ed7654c0f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f66fba063cb4d5599b228308545c515" member_type: VOTER last_known_addr { host: "127.16.10.254" port: 37885 } } peers { permanent_uuid: "ffb833f1426c4a6ea62563ed7654c0f2" member_type: VOTER last_known_addr { host: "127.16.10.253" port: 33517 } } peers { permanent_uuid: "b98590844ccb4749a2a11ebcc6cd60cb" member_type: VOTER last_known_addr { host: "127.16.10.252" port: 36395 } } }
14:00:07.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.091650 16691 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffb833f1426c4a6ea62563ed7654c0f2 [sys.catalog]: This master's current role is: FOLLOWER
14:00:07.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.091962 16691 raft_consensus.cc:493] T 00000000000000000000000000000000 P ffb833f1426c4a6ea62563ed7654c0f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
14:00:07.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.092203 16691 raft_consensus.cc:515] T 00000000000000000000000000000000 P ffb833f1426c4a6ea62563ed7654c0f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f66fba063cb4d5599b228308545c515" member_type: VOTER last_known_addr { host: "127.16.10.254" port: 37885 } } peers { permanent_uuid: "ffb833f1426c4a6ea62563ed7654c0f2" member_type: VOTER last_known_addr { host: "127.16.10.253" port: 33517 } } peers { permanent_uuid: "b98590844ccb4749a2a11ebcc6cd60cb" member_type: VOTER last_known_addr { host: "127.16.10.252" port: 36395 } }
14:00:07.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.093909 16691 leader_election.cc:290] T 00000000000000000000000000000000 P ffb833f1426c4a6ea62563ed7654c0f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f66fba063cb4d5599b228308545c515 (127.16.10.254:37885), b98590844ccb4749a2a11ebcc6cd60cb (127.16.10.252:36395)
14:00:07.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.094249 16684 catalog_manager.cc:6069] T 00000000000000000000000000000000 P 5f66fba063cb4d5599b228308545c515: Generated new TSK 0
14:00:07.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.096228 16479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ffb833f1426c4a6ea62563ed7654c0f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f66fba063cb4d5599b228308545c515" is_pre_election: true
14:00:07.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.096690 16684 catalog_manager.cc:1549] Initializing in-progress tserver states...
14:00:07.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.101576 16621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ffb833f1426c4a6ea62563ed7654c0f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b98590844ccb4749a2a11ebcc6cd60cb" is_pre_election: true
14:00:07.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.102419 16527 leader_election.cc:304] T 00000000000000000000000000000000 P ffb833f1426c4a6ea62563ed7654c0f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ffb833f1426c4a6ea62563ed7654c0f2; no voters: 5f66fba063cb4d5599b228308545c515, b98590844ccb4749a2a11ebcc6cd60cb
14:00:07.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.102988 16692 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ffb833f1426c4a6ea62563ed7654c0f2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
14:00:07.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.104538 16575 sys_catalog.cc:565] T 00000000000000000000000000000000 P ffb833f1426c4a6ea62563ed7654c0f2 [sys.catalog]: configured and running, proceeding with master startup.
14:00:07.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.104624 16550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5f66fba063cb4d5599b228308545c515" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffb833f1426c4a6ea62563ed7654c0f2" is_pre_election: true
14:00:07.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.105139 16550 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ffb833f1426c4a6ea62563ed7654c0f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f66fba063cb4d5599b228308545c515 in term 0.
14:00:07.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.106132 16548 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ffb833f1426c4a6ea62563ed7654c0f2 [term 0 FOLLOWER]: Advancing to term 1
14:00:07.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.109984 16548 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ffb833f1426c4a6ea62563ed7654c0f2 [term 1 FOLLOWER]: Refusing update from remote peer 5f66fba063cb4d5599b228308545c515: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
14:00:07.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.110716 16549 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5f66fba063cb4d5599b228308545c515" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffb833f1426c4a6ea62563ed7654c0f2"
14:00:07.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.111119 16682 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 5f66fba063cb4d5599b228308545c515 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffb833f1426c4a6ea62563ed7654c0f2" member_type: VOTER last_known_addr { host: "127.16.10.253" port: 33517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
14:00:07.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.114110 16549 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ffb833f1426c4a6ea62563ed7654c0f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f66fba063cb4d5599b228308545c515 in term 1.
14:00:07.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.127727 16427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu
14:00:07.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu
14:00:07.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/wal
14:00:07.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/data
14:00:07.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:00:07.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:00:07.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:00:07.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:00:07.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:00:07.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:00:07.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/logs
14:00:07.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/data/info.pb
14:00:07.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:00:07.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:00:07.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:00:07.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:00:07.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:00:07.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:00:07.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:00:07.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:00:07.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:00:07.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:00:07.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.10.193:0
14:00:07.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.10.193
14:00:07.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.10.193
14:00:07.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:00:07.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.10.254:37885,127.16.10.253:33517,127.16.10.252:36395
14:00:07.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.10.212:33969
14:00:07.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:00:07.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:00:07.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:00:07.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.129315 16691 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffb833f1426c4a6ea62563ed7654c0f2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5f66fba063cb4d5599b228308545c515" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f66fba063cb4d5599b228308545c515" member_type: VOTER last_known_addr { host: "127.16.10.254" port: 37885 } } peers { permanent_uuid: "ffb833f1426c4a6ea62563ed7654c0f2" member_type: VOTER last_known_addr { host: "127.16.10.253" port: 33517 } } peers { permanent_uuid: "b98590844ccb4749a2a11ebcc6cd60cb" member_type: VOTER last_known_addr { host: "127.16.10.252" port: 36395 } } }
14:00:07.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.129578 16691 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffb833f1426c4a6ea62563ed7654c0f2 [sys.catalog]: This master's current role is: FOLLOWER
14:00:07.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.129809 16692 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffb833f1426c4a6ea62563ed7654c0f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f66fba063cb4d5599b228308545c515. Latest consensus state: current_term: 1 leader_uuid: "5f66fba063cb4d5599b228308545c515" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f66fba063cb4d5599b228308545c515" member_type: VOTER last_known_addr { host: "127.16.10.254" port: 37885 } } peers { permanent_uuid: "ffb833f1426c4a6ea62563ed7654c0f2" member_type: VOTER last_known_addr { host: "127.16.10.253" port: 33517 } } peers { permanent_uuid: "b98590844ccb4749a2a11ebcc6cd60cb" member_type: VOTER last_known_addr { host: "127.16.10.252" port: 36395 } } }
14:00:07.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.130059 16692 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffb833f1426c4a6ea62563ed7654c0f2 [sys.catalog]: This master's current role is: FOLLOWER
14:00:07.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.171593 16694 mvcc.cc:204] Tried to move back new op lower bound from 7287246057807085568 to 7287246057156583424. Current Snapshot: MvccSnapshot[applied={T|T < 7287246057807085568}]
14:00:07.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.225893 16708 catalog_manager.cc:1294] Loaded cluster ID: 0328278874934e8bbd585811d9ee778d
14:00:07.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.226542 16708 catalog_manager.cc:1587] T 00000000000000000000000000000000 P ffb833f1426c4a6ea62563ed7654c0f2: loading cluster ID for follower catalog manager: success
14:00:07.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.231007 16708 catalog_manager.cc:1609] T 00000000000000000000000000000000 P ffb833f1426c4a6ea62563ed7654c0f2: acquiring CA information for follower catalog manager: success
14:00:07.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.241219 16708 catalog_manager.cc:1637] T 00000000000000000000000000000000 P ffb833f1426c4a6ea62563ed7654c0f2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
14:00:07.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.667097 16682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f66fba063cb4d5599b228308545c515 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5f66fba063cb4d5599b228308545c515" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f66fba063cb4d5599b228308545c515" member_type: VOTER last_known_addr { host: "127.16.10.254" port: 37885 } } peers { permanent_uuid: "ffb833f1426c4a6ea62563ed7654c0f2" member_type: VOTER last_known_addr { host: "127.16.10.253" port: 33517 } } peers { permanent_uuid: "b98590844ccb4749a2a11ebcc6cd60cb" member_type: VOTER last_known_addr { host: "127.16.10.252" port: 36395 } } }
14:00:07.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.667378 16682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f66fba063cb4d5599b228308545c515 [sys.catalog]: This master's current role is: LEADER
14:00:07.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:07.680188 16696 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:00:07.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:07.680518 16696 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:00:07.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:07.680591 16696 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:00:07.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:07.689487 16696 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:00:07.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:07.689669 16696 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.10.193
14:00:07.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.700564 16696 tablet_server_runner.cc:78] Tablet server non-default flags:
14:00:07.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:00:07.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.10.212:33969
14:00:07.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:00:07.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:00:07.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/data
14:00:07.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/wal
14:00:07.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:00:07.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:00:07.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.10.193:0
14:00:07.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:00:07.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:00:07.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:00:07.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/data/info.pb
14:00:07.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.10.193
14:00:07.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:00:07.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.10.254:37885,127.16.10.253:33517,127.16.10.252:36395
14:00:07.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:00:07.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:00:07.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:00:07.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:00:07.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:00:07.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.10.193
14:00:07.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/logs
14:00:07.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:00:07.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:00:07.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:00:07.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:00:07.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:00:07.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
14:00:07.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:00:07.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 13:43:23 UTC on bdcb31816ec0
14:00:07.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11958
14:00:07.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
14:00:07.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.702919 16696 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:00:07.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.705600 16696 file_cache.cc:492] Constructed file cache file cache with capacity 419430
14:00:07.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:07.716650 16717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:00:07.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:07.724424 16718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:00:07.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:07.731917 16720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:00:07.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.732769 16696 server_base.cc:1061] running on GCE node
14:00:07.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.733582 16696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:00:07.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.734921 16696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:00:07.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.736166 16696 hybrid_clock.cc:648] HybridClock initialized: now 1779112807736091 us; error 67 us; skew 500 ppm
14:00:07.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.736495 16696 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:00:07.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.739248 16696 webserver.cc:533] Webserver started at http://127.16.10.193:44113/ using document root <none> and password file <none>
14:00:07.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.740190 16696 fs_manager.cc:362] Metadata directory not provided
14:00:07.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.740289 16696 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:00:07.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.740622 16696 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
14:00:07.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.743290 16696 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/data/instance:
14:00:07.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "52f4ec85820b4a5897f1e992099d8c08"
14:00:07.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:00:07 on dist-test-slave-rq0s"
14:00:07.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.744190 16696 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/wal/instance:
14:00:07.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "52f4ec85820b4a5897f1e992099d8c08"
14:00:07.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:00:07 on dist-test-slave-rq0s"
14:00:07.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.752060 16696 fs_manager.cc:696] Time spent creating directory manager: real 0.007s	user 0.008s	sys 0.000s
14:00:07.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.755896 16726 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:00:07.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.757480 16696 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.004s
14:00:07.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.757669 16696 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/data,/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/wal
14:00:07.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "52f4ec85820b4a5897f1e992099d8c08"
14:00:07.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:00:07 on dist-test-slave-rq0s"
14:00:07.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.757910 16696 fs_report.cc:389] FS layout report
14:00:07.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:00:07.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/wal
14:00:07.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/wal
14:00:07.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/data/data
14:00:07.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:00:07.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:00:07.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:00:07.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:00:07.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:00:07.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:00:07.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:00:07.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:00:07.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:00:07.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:00:07.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:00:07.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.781430 16696 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:00:07.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.782518 16696 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:00:07.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.782819 16696 kserver.cc:163] Server-wide thread pool size limit: 3276
14:00:07.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.784039 16696 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:00:07.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.786204 16696 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
14:00:07.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.786309 16696 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
14:00:07.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.786412 16696 ts_tablet_manager.cc:616] Registered 0 tablets
14:00:07.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.786494 16696 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
14:00:07.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.836970 16696 rpc_server.cc:307] RPC server started. Bound to: 127.16.10.193:46221
14:00:07.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.837073 16838 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.10.193:46221 every 8 connection(s)
14:00:07.839 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.838845 16696 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/data/info.pb
14:00:07.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.842238 16427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu as pid 16696
14:00:07.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.842442 16427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/wal/instance
14:00:07.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.852806 16427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu
14:00:07.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu
14:00:07.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/wal
14:00:07.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/data
14:00:07.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:00:07.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:00:07.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:00:07.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:00:07.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:00:07.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:00:07.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/logs
14:00:07.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/data/info.pb
14:00:07.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:00:07.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:00:07.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:00:07.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:00:07.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:00:07.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:00:07.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:00:07.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:00:07.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:00:07.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:00:07.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.10.194:0
14:00:07.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.10.194
14:00:07.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.10.194
14:00:07.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:00:07.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.10.254:37885,127.16.10.253:33517,127.16.10.252:36395
14:00:07.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.10.212:33969
14:00:07.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:00:07.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:00:07.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:00:07.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.863657 16680 catalog_manager.cc:1294] Loaded cluster ID: 0328278874934e8bbd585811d9ee778d
14:00:07.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.863818 16680 catalog_manager.cc:1587] T 00000000000000000000000000000000 P b98590844ccb4749a2a11ebcc6cd60cb: loading cluster ID for follower catalog manager: success
14:00:07.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.869716 16680 catalog_manager.cc:1609] T 00000000000000000000000000000000 P b98590844ccb4749a2a11ebcc6cd60cb: acquiring CA information for follower catalog manager: success
14:00:07.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.875880 16680 catalog_manager.cc:1637] T 00000000000000000000000000000000 P b98590844ccb4749a2a11ebcc6cd60cb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
14:00:07.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.879361 16839 heartbeater.cc:344] Connected to a master server at 127.16.10.252:36395
14:00:07.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.879859 16839 heartbeater.cc:461] Registering TS with master...
14:00:07.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.881155 16839 heartbeater.cc:507] Master 127.16.10.252:36395 requested a full tablet report, sending...
14:00:07.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.882251 16840 heartbeater.cc:344] Connected to a master server at 127.16.10.254:37885
14:00:07.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.882417 16840 heartbeater.cc:461] Registering TS with master...
14:00:07.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.883085 16840 heartbeater.cc:507] Master 127.16.10.254:37885 requested a full tablet report, sending...
14:00:07.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.884465 16611 ts_manager.cc:194] Registered new tserver with Master: 52f4ec85820b4a5897f1e992099d8c08 (127.16.10.193:46221)
14:00:07.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.884793 16468 ts_manager.cc:194] Registered new tserver with Master: 52f4ec85820b4a5897f1e992099d8c08 (127.16.10.193:46221)
14:00:07.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.886807 16468 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.16.10.193:60401
14:00:07.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.887238 16841 heartbeater.cc:344] Connected to a master server at 127.16.10.253:33517
14:00:07.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.887405 16841 heartbeater.cc:461] Registering TS with master...
14:00:07.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.888015 16841 heartbeater.cc:507] Master 127.16.10.253:33517 requested a full tablet report, sending...
14:00:07.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:07.889897 16540 ts_manager.cc:194] Registered new tserver with Master: 52f4ec85820b4a5897f1e992099d8c08 (127.16.10.193:46221)
14:00:08.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:08.227586 16847 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:00:08.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:08.227922 16847 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:00:08.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:08.227998 16847 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:00:08.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:08.236831 16847 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:00:08.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:08.237016 16847 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.10.194
14:00:08.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.247455 16847 tablet_server_runner.cc:78] Tablet server non-default flags:
14:00:08.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:00:08.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.10.212:33969
14:00:08.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:00:08.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:00:08.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/data
14:00:08.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/wal
14:00:08.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:00:08.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:00:08.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.10.194:0
14:00:08.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:00:08.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:00:08.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:00:08.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/data/info.pb
14:00:08.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.10.194
14:00:08.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:00:08.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.10.254:37885,127.16.10.253:33517,127.16.10.252:36395
14:00:08.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:00:08.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:00:08.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:00:08.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:00:08.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:00:08.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.10.194
14:00:08.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/logs
14:00:08.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:00:08.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:00:08.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:00:08.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:00:08.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:00:08.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
14:00:08.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:00:08.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 13:43:23 UTC on bdcb31816ec0
14:00:08.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11958
14:00:08.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
14:00:08.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.249351 16847 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:00:08.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.251647 16847 file_cache.cc:492] Constructed file cache file cache with capacity 419430
14:00:08.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:08.262290 16854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:00:08.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:08.262390 16855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:00:08.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:08.268028 16857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:00:08.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.269713 16847 server_base.cc:1061] running on GCE node
14:00:08.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.270434 16847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:00:08.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.271761 16847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:00:08.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.272986 16847 hybrid_clock.cc:648] HybridClock initialized: now 1779112808272947 us; error 29 us; skew 500 ppm
14:00:08.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.273311 16847 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:00:08.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.275722 16847 webserver.cc:533] Webserver started at http://127.16.10.194:33241/ using document root <none> and password file <none>
14:00:08.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.276500 16847 fs_manager.cc:362] Metadata directory not provided
14:00:08.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.276592 16847 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:00:08.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.276911 16847 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
14:00:08.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.279461 16847 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/data/instance:
14:00:08.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "ad837748684c4c93a572185eda10fa14"
14:00:08.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:00:08 on dist-test-slave-rq0s"
14:00:08.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.280195 16847 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/wal/instance:
14:00:08.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "ad837748684c4c93a572185eda10fa14"
14:00:08.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:00:08 on dist-test-slave-rq0s"
14:00:08.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.286188 16847 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.006s	sys 0.000s
14:00:08.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.290043 16863 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:00:08.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.291743 16847 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
14:00:08.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.291968 16847 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/data,/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/wal
14:00:08.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "ad837748684c4c93a572185eda10fa14"
14:00:08.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:00:08 on dist-test-slave-rq0s"
14:00:08.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.292286 16847 fs_report.cc:389] FS layout report
14:00:08.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:00:08.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/wal
14:00:08.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/wal
14:00:08.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/data/data
14:00:08.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:00:08.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:00:08.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:00:08.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:00:08.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:00:08.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:00:08.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:00:08.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:00:08.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:00:08.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:00:08.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:00:08.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.315469 16847 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:00:08.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.316573 16847 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:00:08.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.316927 16847 kserver.cc:163] Server-wide thread pool size limit: 3276
14:00:08.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.318161 16847 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:00:08.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.320065 16847 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
14:00:08.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.320199 16847 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
14:00:08.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.320312 16847 ts_tablet_manager.cc:616] Registered 0 tablets
14:00:08.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.320390 16847 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
14:00:08.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.359143 16847 rpc_server.cc:307] RPC server started. Bound to: 127.16.10.194:44603
14:00:08.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.359234 16975 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.10.194:44603 every 8 connection(s)
14:00:08.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.360841 16847 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/data/info.pb
14:00:08.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.368947 16427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu as pid 16847
14:00:08.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.369151 16427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/wal/instance
14:00:08.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.373410 16427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu
14:00:08.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu
14:00:08.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/wal
14:00:08.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/data
14:00:08.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:00:08.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:00:08.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:00:08.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:00:08.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:00:08.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:00:08.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/logs
14:00:08.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/data/info.pb
14:00:08.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:00:08.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:00:08.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:00:08.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:00:08.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:00:08.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:00:08.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:00:08.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:00:08.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:00:08.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:00:08.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.10.195:0
14:00:08.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.10.195
14:00:08.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.10.195
14:00:08.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:00:08.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.10.254:37885,127.16.10.253:33517,127.16.10.252:36395
14:00:08.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.10.212:33969
14:00:08.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:00:08.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:00:08.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:00:08.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.383720 16976 heartbeater.cc:344] Connected to a master server at 127.16.10.252:36395
14:00:08.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.383798 16978 heartbeater.cc:344] Connected to a master server at 127.16.10.253:33517
14:00:08.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.384179 16976 heartbeater.cc:461] Registering TS with master...
14:00:08.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.384184 16978 heartbeater.cc:461] Registering TS with master...
14:00:08.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.385509 16978 heartbeater.cc:507] Master 127.16.10.253:33517 requested a full tablet report, sending...
14:00:08.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.385543 16976 heartbeater.cc:507] Master 127.16.10.252:36395 requested a full tablet report, sending...
14:00:08.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.385607 16977 heartbeater.cc:344] Connected to a master server at 127.16.10.254:37885
14:00:08.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.385779 16977 heartbeater.cc:461] Registering TS with master...
14:00:08.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.386338 16977 heartbeater.cc:507] Master 127.16.10.254:37885 requested a full tablet report, sending...
14:00:08.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.387741 16468 ts_manager.cc:194] Registered new tserver with Master: ad837748684c4c93a572185eda10fa14 (127.16.10.194:44603)
14:00:08.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.387722 16611 ts_manager.cc:194] Registered new tserver with Master: ad837748684c4c93a572185eda10fa14 (127.16.10.194:44603)
14:00:08.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.387864 16540 ts_manager.cc:194] Registered new tserver with Master: ad837748684c4c93a572185eda10fa14 (127.16.10.194:44603)
14:00:08.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.388811 16468 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.16.10.194:36927
14:00:08.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:08.724300 16983 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:00:08.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:08.724644 16983 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:00:08.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:08.724720 16983 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:00:08.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:08.733525 16983 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:00:08.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:08.733705 16983 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.10.195
14:00:08.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.744021 16983 tablet_server_runner.cc:78] Tablet server non-default flags:
14:00:08.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:00:08.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.10.212:33969
14:00:08.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:00:08.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:00:08.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/data
14:00:08.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/wal
14:00:08.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:00:08.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:00:08.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.10.195:0
14:00:08.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:00:08.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:00:08.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:00:08.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/data/info.pb
14:00:08.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.10.195
14:00:08.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:00:08.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.10.254:37885,127.16.10.253:33517,127.16.10.252:36395
14:00:08.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:00:08.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:00:08.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:00:08.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:00:08.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:00:08.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.10.195
14:00:08.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/logs
14:00:08.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:00:08.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:00:08.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:00:08.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:00:08.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:00:08.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
14:00:08.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:00:08.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 13:43:23 UTC on bdcb31816ec0
14:00:08.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11958
14:00:08.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
14:00:08.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.745589 16983 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:00:08.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.747509 16983 file_cache.cc:492] Constructed file cache file cache with capacity 419430
14:00:08.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:08.757504 16988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:00:08.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:08.757481 16989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:00:08.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:08.759142 16991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:00:08.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.759763 16983 server_base.cc:1061] running on GCE node
14:00:08.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.760473 16983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:00:08.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.761587 16983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:00:08.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.762818 16983 hybrid_clock.cc:648] HybridClock initialized: now 1779112808762750 us; error 60 us; skew 500 ppm
14:00:08.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.763185 16983 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:00:08.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.766095 16983 webserver.cc:533] Webserver started at http://127.16.10.195:45985/ using document root <none> and password file <none>
14:00:08.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.767187 16983 fs_manager.cc:362] Metadata directory not provided
14:00:08.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.767315 16983 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:00:08.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.767704 16983 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
14:00:08.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.770977 16983 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/data/instance:
14:00:08.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "948ae7bedcae4f2ea8a1825119425668"
14:00:08.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:00:08 on dist-test-slave-rq0s"
14:00:08.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.771967 16983 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/wal/instance:
14:00:08.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "948ae7bedcae4f2ea8a1825119425668"
14:00:08.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:00:08 on dist-test-slave-rq0s"
14:00:08.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.777689 16983 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
14:00:08.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.781342 16997 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:00:08.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.783022 16983 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
14:00:08.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.783174 16983 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/data,/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/wal
14:00:08.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "948ae7bedcae4f2ea8a1825119425668"
14:00:08.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:00:08 on dist-test-slave-rq0s"
14:00:08.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.783361 16983 fs_report.cc:389] FS layout report
14:00:08.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:00:08.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/wal
14:00:08.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/wal
14:00:08.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/data/data
14:00:08.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:00:08.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:00:08.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:00:08.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:00:08.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:00:08.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:00:08.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:00:08.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:00:08.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:00:08.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:00:08.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:00:08.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.807042 16983 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:00:08.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.808061 16983 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:00:08.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.808406 16983 kserver.cc:163] Server-wide thread pool size limit: 3276
14:00:08.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.809525 16983 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:00:08.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.811339 16983 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
14:00:08.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.811442 16983 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
14:00:08.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.811522 16983 ts_tablet_manager.cc:616] Registered 0 tablets
14:00:08.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.811568 16983 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
14:00:08.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.848244 16983 rpc_server.cc:307] RPC server started. Bound to: 127.16.10.195:40871
14:00:08.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.848345 17109 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.10.195:40871 every 8 connection(s)
14:00:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.849689 16983 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/data/info.pb
14:00:08.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.850792 16427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu as pid 16983
14:00:08.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.850992 16427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/wal/instance
14:00:08.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.869344 17112 heartbeater.cc:344] Connected to a master server at 127.16.10.253:33517
14:00:08.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.869419 17111 heartbeater.cc:344] Connected to a master server at 127.16.10.254:37885
14:00:08.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.869376 17110 heartbeater.cc:344] Connected to a master server at 127.16.10.252:36395
14:00:08.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.869884 17110 heartbeater.cc:461] Registering TS with master...
14:00:08.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.869879 17112 heartbeater.cc:461] Registering TS with master...
14:00:08.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.869879 17111 heartbeater.cc:461] Registering TS with master...
14:00:08.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.871162 17112 heartbeater.cc:507] Master 127.16.10.253:33517 requested a full tablet report, sending...
14:00:08.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.871222 17111 heartbeater.cc:507] Master 127.16.10.254:37885 requested a full tablet report, sending...
14:00:08.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.871193 17110 heartbeater.cc:507] Master 127.16.10.252:36395 requested a full tablet report, sending...
14:00:08.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.873037 16468 ts_manager.cc:194] Registered new tserver with Master: 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871)
14:00:08.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.873245 16611 ts_manager.cc:194] Registered new tserver with Master: 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871)
14:00:08.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.873872 16540 ts_manager.cc:194] Registered new tserver with Master: 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871)
14:00:08.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.874171 16468 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.16.10.195:52201
14:00:08.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.878364 16427 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
14:00:08.880 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
14:00:08.881 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_masters {
}

14:00:08.891 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_masters {
  masters {
    id {
      type: MASTER
      index: 0
    }
    bound_rpc_address {
      host: "127.16.10.254"
      port: 37885
    }
    bound_http_address {
      host: "127.16.10.254"
      port: 35859
    }
  }
  masters {
    id {
      type: MASTER
      index: 1
    }
    bound_rpc_address {
      host: "127.16.10.253"
      port: 33517
    }
    bound_http_address {
      host: "127.16.10.253"
      port: 43955
    }
  }
  masters {
    id {
      type: MASTER
      index: 2
    }
    bound_rpc_address {
      host: "127.16.10.252"
      port: 36395
    }
    bound_http_address {
      host: "127.16.10.252"
      port: 43585
    }
  }
}

14:00:08.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:08.890635 16840 heartbeater.cc:499] Master 127.16.10.254:37885 was elected leader, sending a full tablet report...
14:00:08.895 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_tservers {
}

14:00:08.897 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_tservers {
  tservers {
    id {
      type: TSERVER
      index: 0
    }
    bound_rpc_address {
      host: "127.16.10.193"
      port: 46221
    }
    bound_http_address {
      host: "127.16.10.193"
      port: 44113
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 1
    }
    bound_rpc_address {
      host: "127.16.10.194"
      port: 44603
    }
    bound_http_address {
      host: "127.16.10.194"
      port: 33241
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 2
    }
    bound_rpc_address {
      host: "127.16.10.195"
      port: 40871
    }
    bound_http_address {
      host: "127.16.10.195"
      port: 45985
    }
  }
}

14:00:08.898 [INFO - main] (KuduTestHarness.java:157) Creating a new Kudu client...
14:00:09.066 [DEBUG - main] (CompositeStrategy.java:59) Unable to load modern Subject methods, relying only on legacy methods
java.lang.NoSuchMethodException: javax.security.auth.Subject.current()
	at java.base/java.lang.Class.getDeclaredMethod(Class.java:2677)
	at org.apache.kudu.client.internals.ModernStrategy.<init>(ModernStrategy.java:51)
	at org.apache.kudu.client.internals.CompositeStrategy.<init>(CompositeStrategy.java:52)
	at org.apache.kudu.client.internals.CompositeStrategy.<clinit>(CompositeStrategy.java:41)
	at org.apache.kudu.client.internals.SecurityManagerCompatibility.get(SecurityManagerCompatibility.java:43)
	at org.apache.kudu.client.SecurityContext.setupSubject(SecurityContext.java:155)
	at org.apache.kudu.client.SecurityContext.<init>(SecurityContext.java:138)
	at org.apache.kudu.client.AsyncKuduClient.<init>(AsyncKuduClient.java:431)
	at org.apache.kudu.client.AsyncKuduClient$AsyncKuduClientBuilder.build(AsyncKuduClient.java:3109)
	at org.apache.kudu.test.KuduTestHarness.before(KuduTestHarness.java:161)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:50)
	at org.apache.kudu.test.junit.RetryRule$RetryStatement.doOneAttemptAndReport(RetryRule.java:184)
	at org.apache.kudu.test.junit.RetryRule$RetryStatement.evaluate(RetryRule.java:244)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:27)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
	at org.junit.runner.JUnitCore.runMain(JUnitCore.java:77)
	at org.junit.runner.JUnitCore.main(JUnitCore.java:36)
14:00:09.115 [DEBUG - main] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 
14:00:09.303 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.16.10.252:36395(127.16.10.252:36395)] connecting to peer
14:00:09.354 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.16.10.253:33517(127.16.10.253:33517)] connecting to peer
14:00:09.356 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.16.10.254:37885(127.16.10.254:37885)] connecting to peer
14:00:09.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:09.391152 16977 heartbeater.cc:499] Master 127.16.10.254:37885 was elected leader, sending a full tablet report...
14:00:09.394 [DEBUG - kudu-nio-2] (Connection.java:757) [peer master-127.16.10.254:37885(127.16.10.254:37885)] Successfully connected to peer
14:00:09.402 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.16.10.252:36395(127.16.10.252:36395)] Successfully connected to peer
14:00:09.431 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.16.10.253:33517(127.16.10.253:33517)] Successfully connected to peer
14:00:09.497 [DEBUG - kudu-nio-0] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.16.10.252
14:00:09.497 [DEBUG - kudu-nio-1] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.16.10.253
14:00:09.499 [DEBUG - kudu-nio-2] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.16.10.254
14:00:09.726 [DEBUG - kudu-nio-2] (Negotiator.java:919) Authenticated connection [id: 0xda3dc4b9, L:/127.0.0.1:60836 - R:/127.16.10.254:37885] using SASL/PLAIN
14:00:09.726 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0xac431598, L:/127.0.0.1:34408 - R:/127.16.10.253:33517] using SASL/PLAIN
14:00:09.727 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x80b273eb, L:/127.0.0.1:58512 - R:/127.16.10.252:36395] using SASL/PLAIN
14:00:09.769 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet Kudu Master for table 'Kudu Master' with partition [<start>, <end>)
14:00:09.771 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=CreateTable, tablet=null, attempt=1, TimeoutTracker(timeout=50000, elapsed=511), Traces: [0ms] refreshing cache from master, [27ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.10.252:36395, [89ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.10.253:33517, [91ms] Sub RPC ConnectToMaster: sending RPC to server master-127.16.10.254:37885, [478ms] Sub RPC ConnectToMaster: received response from server master-127.16.10.253:33517: OK, [478ms] Sub RPC ConnectToMaster: received response from server master-127.16.10.252:36395: OK, [478ms] Sub RPC ConnectToMaster: received response from server master-127.16.10.254:37885: OK, deferred=null) after lookup
14:00:09.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:09.827322 16468 catalog_manager.cc:2282] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60836:
14:00:09.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) name: "org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244"
14:00:09.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) schema {
14:00:09.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
14:00:09.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "key"
14:00:09.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
14:00:09.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: true
14:00:09.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
14:00:09.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
14:00:09.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
14:00:09.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
14:00:09.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
14:00:09.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
14:00:09.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column1_i"
14:00:09.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
14:00:09.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
14:00:09.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
14:00:09.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
14:00:09.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
14:00:09.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
14:00:09.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
14:00:09.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
14:00:09.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column2_i"
14:00:09.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
14:00:09.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
14:00:09.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
14:00:09.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
14:00:09.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
14:00:09.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
14:00:09.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
14:00:09.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
14:00:09.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column3_s"
14:00:09.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: STRING
14:00:09.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
14:00:09.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: true
14:00:09.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     encoding: DICT_ENCODING
14:00:09.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     compression: LZ4
14:00:09.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 4096
14:00:09.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
14:00:09.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
14:00:09.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
14:00:09.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
14:00:09.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column4_b"
14:00:09.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: BOOL
14:00:09.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
14:00:09.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
14:00:09.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
14:00:09.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
14:00:09.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
14:00:09.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
14:00:09.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
14:00:09.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) partition_schema {
14:00:09.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   range_schema {
14:00:09.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     columns {
14:00:09.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)       name: "key"
14:00:09.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     }
14:00:09.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
14:00:09.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
14:00:09.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:09.829491 16468 catalog_manager.cc:7058] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
14:00:09.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:09.877177 17111 heartbeater.cc:499] Master 127.16.10.254:37885 was elected leader, sending a full tablet report...
14:00:09.913 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:894) Opened table 7c1c64d2643043cb9e53fcc9b3648575
14:00:09.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:09.924183 17045 tablet_service.cc:1511] Processing CreateTablet for tablet cf8e0efa2228413d8e2421c76198d284 (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244 [id=7c1c64d2643043cb9e53fcc9b3648575]), partition=RANGE (key) PARTITION UNBOUNDED
14:00:09.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:09.925976 17045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf8e0efa2228413d8e2421c76198d284. 1 dirs total, 0 dirs full, 0 dirs failed
14:00:09.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:09.930424 16774 tablet_service.cc:1511] Processing CreateTablet for tablet cf8e0efa2228413d8e2421c76198d284 (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244 [id=7c1c64d2643043cb9e53fcc9b3648575]), partition=RANGE (key) PARTITION UNBOUNDED
14:00:09.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:09.935331 16911 tablet_service.cc:1511] Processing CreateTablet for tablet cf8e0efa2228413d8e2421c76198d284 (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244 [id=7c1c64d2643043cb9e53fcc9b3648575]), partition=RANGE (key) PARTITION UNBOUNDED
14:00:09.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:09.936885 16911 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf8e0efa2228413d8e2421c76198d284. 1 dirs total, 0 dirs full, 0 dirs failed
14:00:09.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:09.940595 16774 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf8e0efa2228413d8e2421c76198d284. 1 dirs total, 0 dirs full, 0 dirs failed
14:00:09.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:09.952191 17144 tablet_bootstrap.cc:492] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668: Bootstrap starting.
14:00:09.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:09.957674 17144 tablet_bootstrap.cc:654] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668: Neither blocks nor log segments found. Creating new log.
14:00:09.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:09.960134 17145 tablet_bootstrap.cc:492] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08: Bootstrap starting.
14:00:09.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:09.960206 17144 log.cc:826] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668: Log is configured to *not* fsync() on all Append() calls
14:00:09.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:09.965323 17146 tablet_bootstrap.cc:492] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14: Bootstrap starting.
14:00:09.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:09.969480 17146 tablet_bootstrap.cc:654] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14: Neither blocks nor log segments found. Creating new log.
14:00:09.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:09.969480 17145 tablet_bootstrap.cc:654] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08: Neither blocks nor log segments found. Creating new log.
14:00:09.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:09.970943 17145 log.cc:826] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08: Log is configured to *not* fsync() on all Append() calls
14:00:09.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:09.971410 17146 log.cc:826] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14: Log is configured to *not* fsync() on all Append() calls
14:00:09.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:09.977072 17144 tablet_bootstrap.cc:492] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668: No bootstrap required, opened a new log
14:00:09.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:09.977159 17146 tablet_bootstrap.cc:492] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14: No bootstrap required, opened a new log
14:00:09.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:09.977460 17146 ts_tablet_manager.cc:1403] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14: Time spent bootstrapping tablet: real 0.012s	user 0.006s	sys 0.003s
14:00:09.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:09.977550 17145 tablet_bootstrap.cc:492] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08: No bootstrap required, opened a new log
14:00:09.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:09.977761 17144 ts_tablet_manager.cc:1403] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668: Time spent bootstrapping tablet: real 0.026s	user 0.002s	sys 0.011s
14:00:09.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:09.977795 17145 ts_tablet_manager.cc:1403] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08: Time spent bootstrapping tablet: real 0.018s	user 0.011s	sys 0.000s
14:00:09.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:09.984340 17146 raft_consensus.cc:359] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:09.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:09.985065 17146 raft_consensus.cc:385] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
14:00:09.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:09.985085 17144 raft_consensus.cc:359] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:09.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:09.985462 17146 raft_consensus.cc:740] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad837748684c4c93a572185eda10fa14, State: Initialized, Role: FOLLOWER
14:00:09.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:09.985774 17144 raft_consensus.cc:385] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
14:00:09.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:09.986037 17144 raft_consensus.cc:740] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948ae7bedcae4f2ea8a1825119425668, State: Initialized, Role: FOLLOWER
14:00:09.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:09.985899 17145 raft_consensus.cc:359] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:09.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:09.986301 17145 raft_consensus.cc:385] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
14:00:09.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:09.986410 17145 raft_consensus.cc:740] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f4ec85820b4a5897f1e992099d8c08, State: Initialized, Role: FOLLOWER
14:00:09.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:09.986308 17146 consensus_queue.cc:260] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:09.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:09.987147 17145 consensus_queue.cc:260] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:09.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:09.987797 17144 consensus_queue.cc:260] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:09.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:09.993947 17144 ts_tablet_manager.cc:1434] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668: Time spent starting tablet: real 0.016s	user 0.010s	sys 0.002s
14:00:09.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:09.994692 17146 ts_tablet_manager.cc:1434] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14: Time spent starting tablet: real 0.017s	user 0.012s	sys 0.004s
14:00:10.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:10.012341 17145 ts_tablet_manager.cc:1434] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08: Time spent starting tablet: real 0.034s	user 0.024s	sys 0.009s
14:00:10.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:10.106746 17113 tablet.cc:2404] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668: Can't schedule compaction. Clean time has not been advanced past its initial value.
14:00:10.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:10.112407 16842 tablet.cc:2404] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08: Can't schedule compaction. Clean time has not been advanced past its initial value.
14:00:10.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:10.120296 16979 tablet.cc:2404] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14: Can't schedule compaction. Clean time has not been advanced past its initial value.
14:00:10.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:10.196484 17152 raft_consensus.cc:493] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
14:00:10.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:10.196820 17152 raft_consensus.cc:515] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:10.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:10.198880 17152 leader_election.cc:290] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad837748684c4c93a572185eda10fa14 (127.16.10.194:44603), 52f4ec85820b4a5897f1e992099d8c08 (127.16.10.193:46221)
14:00:10.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:10.205900 16931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf8e0efa2228413d8e2421c76198d284" candidate_uuid: "948ae7bedcae4f2ea8a1825119425668" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad837748684c4c93a572185eda10fa14" is_pre_election: true
14:00:10.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:10.206880 16931 raft_consensus.cc:2468] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948ae7bedcae4f2ea8a1825119425668 in term 0.
14:00:10.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:10.207346 16794 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf8e0efa2228413d8e2421c76198d284" candidate_uuid: "948ae7bedcae4f2ea8a1825119425668" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f4ec85820b4a5897f1e992099d8c08" is_pre_election: true
14:00:10.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:10.207764 17000 leader_election.cc:304] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948ae7bedcae4f2ea8a1825119425668, ad837748684c4c93a572185eda10fa14; no voters: 
14:00:10.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:10.207937 16794 raft_consensus.cc:2468] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948ae7bedcae4f2ea8a1825119425668 in term 0.
14:00:10.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:10.208258 17152 raft_consensus.cc:2804] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 0 FOLLOWER]: Leader pre-election won for term 1
14:00:10.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:10.208417 17152 raft_consensus.cc:493] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
14:00:10.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:10.208506 17152 raft_consensus.cc:3060] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 0 FOLLOWER]: Advancing to term 1
14:00:10.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:10.210896 17152 raft_consensus.cc:515] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:10.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:10.211719 17152 leader_election.cc:290] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [CANDIDATE]: Term 1 election: Requested vote from peers ad837748684c4c93a572185eda10fa14 (127.16.10.194:44603), 52f4ec85820b4a5897f1e992099d8c08 (127.16.10.193:46221)
14:00:10.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:10.212384 16931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf8e0efa2228413d8e2421c76198d284" candidate_uuid: "948ae7bedcae4f2ea8a1825119425668" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad837748684c4c93a572185eda10fa14"
14:00:10.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:10.212381 16794 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf8e0efa2228413d8e2421c76198d284" candidate_uuid: "948ae7bedcae4f2ea8a1825119425668" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f4ec85820b4a5897f1e992099d8c08"
14:00:10.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:10.212642 16794 raft_consensus.cc:3060] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 0 FOLLOWER]: Advancing to term 1
14:00:10.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:10.212607 16931 raft_consensus.cc:3060] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 0 FOLLOWER]: Advancing to term 1
14:00:10.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:10.214947 16794 raft_consensus.cc:2468] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948ae7bedcae4f2ea8a1825119425668 in term 1.
14:00:10.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:10.215225 16931 raft_consensus.cc:2468] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948ae7bedcae4f2ea8a1825119425668 in term 1.
14:00:10.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:10.215523 16999 leader_election.cc:304] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f4ec85820b4a5897f1e992099d8c08, 948ae7bedcae4f2ea8a1825119425668; no voters: 
14:00:10.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:10.215868 17152 raft_consensus.cc:2804] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 1 FOLLOWER]: Leader election won for term 1
14:00:10.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:10.216346 17152 raft_consensus.cc:697] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 1 LEADER]: Becoming Leader. State: Replica: 948ae7bedcae4f2ea8a1825119425668, State: Running, Role: LEADER
14:00:10.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:10.216915 17152 consensus_queue.cc:237] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:10.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:10.222967 16469 catalog_manager.cc:5696] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 reported cstate change: term changed from 0 to 1, leader changed from <none> to 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195). New cstate: current_term: 1 leader_uuid: "948ae7bedcae4f2ea8a1825119425668" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } health_report { overall_health: UNKNOWN } } }
14:00:10.290 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:894) Opened table 7c1c64d2643043cb9e53fcc9b3648575
14:00:10.311 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:10.317 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299994, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221),948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)[L],ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)]}]
14:00:10.318 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=null, attempt=1, sequence_id=-1, TimeoutTracker(timeout=30000, elapsed=18), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [10ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=null) after lookup
14:00:10.327 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] connecting to peer
14:00:10.330 [DEBUG - kudu-nio-3] (Connection.java:757) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] Successfully connected to peer
14:00:10.424 [DEBUG - kudu-nio-3] (Negotiator.java:919) Authenticated connection [id: 0x0614f619, L:/127.0.0.1:38772 - R:/127.16.10.195:40871] using TOKEN/null
14:00:10.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:10.440346 16931 raft_consensus.cc:1275] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 1 FOLLOWER]: Refusing update from remote peer 948ae7bedcae4f2ea8a1825119425668: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
14:00:10.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:10.440420 16794 raft_consensus.cc:1275] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 1 FOLLOWER]: Refusing update from remote peer 948ae7bedcae4f2ea8a1825119425668: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
14:00:10.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:10.441659 17155 consensus_queue.cc:1048] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
14:00:10.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:10.441985 17152 consensus_queue.cc:1048] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
14:00:10.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:10.462397 17159 mvcc.cc:204] Tried to move back new op lower bound from 7287246071551328256 to 7287246070652547072. Current Snapshot: MvccSnapshot[applied={T|T < 7287246071551328256 or (T in {7287246071551328256})}]
14:00:10.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:10.492754 17160 mvcc.cc:204] Tried to move back new op lower bound from 7287246071551328256 to 7287246070652547072. Current Snapshot: MvccSnapshot[applied={T|T < 7287246071551328256}]
14:00:10.589 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 3 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244, tablet=cf8e0efa2228413d8e2421c76198d284, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:00:10.591 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.16.10.195:40871
14:00:10.593 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

14:00:10.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:10.594053 16427 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu pid 16983
14:00:10.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:10.605275 17105 generic_service.cc:196] Checking for leaks (request via RPC)
14:00:10.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:10.707648 16427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu with pid 16983
14:00:10.733 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:00:10.737 [DEBUG - kudu-nio-3] (Connection.java:239) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] handling channelInactive
14:00:10.737 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] cleaning up while in state READY due to: connection closed
14:00:10.738 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871) for tablet cf8e0efa2228413d8e2421c76198d284: connection closed
14:00:10.745 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:10.745 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221),948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)[L],ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)]}]
14:00:10.746 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=2, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=12), Traces: [0ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [5ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: connection closed, [5ms] delaying RPC due to: Network error: connection closed, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [12ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@1711764220(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@76ef25a0 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6f803814 -> wakeup thread Time-limited test)) after lookup
14:00:10.747 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] connecting to peer
14:00:10.750 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:10.750 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] cleaning up while in state CONNECTING due to: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:10.750 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:10.765 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:10.765 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221),948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)[L],ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)]}]
14:00:10.766 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=4, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=32), Traces: [0ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [5ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: connection closed, [5ms] delaying RPC due to: Network error: connection closed, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [12ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [13ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [17ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [17ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [31ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@1711764220(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@76ef25a0 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1561206150 after retry RPC@1025970902), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6f803814 -> wakeup thread Time-limited test -> (continuation of Deferred@1561206150 after retry RPC@1025970902))) after lookup
14:00:10.767 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] connecting to peer
14:00:10.769 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:10.769 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] cleaning up while in state CONNECTING due to: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:10.769 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:10.805 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:10.805 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221),948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)[L],ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)]}]
14:00:10.806 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=6, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=72), Traces: [0ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [5ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: connection closed, [5ms] delaying RPC due to: Network error: connection closed, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [12ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [13ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [17ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [17ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [31ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [33ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [36ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [36ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [68ms] refreshing cache from master, [68ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [71ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@1711764220(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@76ef25a0 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1561206150 after retry RPC@1025970902) -> (continuation of Deferred@1978772652 after retry RPC@976911949), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6f803814 -> wakeup thread Time-limited test -> (continuation of Deferred@1561206150 after retry RPC@1025970902) -> (continuation of Deferred@1978772652 after retry RPC@976911949))) after lookup
14:00:10.807 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] connecting to peer
14:00:10.809 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:10.809 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] cleaning up while in state CONNECTING due to: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:10.809 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:10.906 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:10.906 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221),948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)[L],ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)]}]
14:00:10.907 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=8, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=173), Traces: [0ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [5ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: connection closed, [5ms] delaying RPC due to: Network error: connection closed, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [12ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [13ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [17ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [17ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [31ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [33ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [36ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [36ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [68ms] refreshing cache from master, [68ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [71ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [73ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [76ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [76ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [168ms] refreshing cache from master, [169ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [172ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@1711764220(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@76ef25a0 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1561206150 after retry RPC@1025970902) -> (continuation of Deferred@1978772652 after retry RPC@976911949) -> (continuation of Deferred@2026605856 after retry RPC@1261405468), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6f803814 -> wakeup thread Time-limited test -> (continuation of Deferred@1561206150 after retry RPC@1025970902) -> (continuation of Deferred@1978772652 after retry RPC@976911949) -> (continuation of Deferred@2026605856 after retry RPC@1261405468))) after lookup
14:00:10.908 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] connecting to peer
14:00:10.910 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:10.910 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] cleaning up while in state CONNECTING due to: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:10.911 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:11.225 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:11.226 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221),948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)[L],ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)]}]
14:00:11.227 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=10, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=493), Traces: [0ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [5ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: connection closed, [5ms] delaying RPC due to: Network error: connection closed, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [12ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [13ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [17ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [17ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [31ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [33ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [36ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [36ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [68ms] refreshing cache from master, [68ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [71ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [73ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [76ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [76ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [168ms] refreshing cache from master, [169ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [172ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [175ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [178ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [178ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [488ms] refreshing cache from master, [488ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [492ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@1711764220(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@76ef25a0 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1561206150 after retry RPC@1025970902) -> (continuation of Deferred@1978772652 after retry RPC@976911949) -> (continuation of Deferred@2026605856 after retry RPC@1261405468) -> (continuation of Deferred@1111868658 after retry RPC@1358276775), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6f803814 -> wakeup thread Time-limited test -> (continuation of Deferred@1561206150 after retry RPC@1025970902) -> (continuation of Deferred@1978772652 after retry RPC@976911949) -> (continuation of Deferred@2026605856 after retry RPC@1261405468) -> (continuation of Deferred@1111868658 after retry RPC@1358276775))) after lookup
14:00:11.227 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] connecting to peer
14:00:11.229 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:11.229 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] cleaning up while in state CONNECTING due to: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:11.230 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:11.408 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:11.409 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221),948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)[L],ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)]}]
14:00:11.409 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=12, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=675), Traces: [0ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [5ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: connection closed, [5ms] delaying RPC due to: Network error: connection closed, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [12ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [13ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [17ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [17ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [31ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [33ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [36ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [36ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [68ms] refreshing cache from master, [68ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [71ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [73ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [76ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [76ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [168ms] refreshing cache from master, [169ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [172ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [175ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [178ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [178ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [488ms] refreshing cache from master, [488ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [492ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [494ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [497ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [497ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [668ms] refreshing cache from master, [668ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [674ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@1711764220(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@76ef25a0 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1561206150 after retry RPC@1025970902) -> (continuation of Deferred@1978772652 after retry RPC@976911949) -> (continuation of Deferred@2026605856 after retry RPC@1261405468) -> (continuation of Deferred@1111868658 after retry RPC@1358276775) -> (continuation of Deferred@692357164 after retry RPC@1409920307), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6f803814 -> wakeup thread Time-limited test -> (continuation of Deferred@1561206150 after retry RPC@1025970902) -> (continuation of Deferred@1978772652 after retry RPC@976911949) -> (continuation of Deferred@2026605856 after retry RPC@1261405468) -> (continuation of Deferred@1111868658 after retry RPC@1358276775) -> (continuation of Deferred@692357164 after retry RPC@1409920307))) after lookup
14:00:11.410 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] connecting to peer
14:00:11.412 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:11.412 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] cleaning up while in state CONNECTING due to: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:11.412 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:12.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:12.017122 17176 raft_consensus.cc:493] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948ae7bedcae4f2ea8a1825119425668)
14:00:12.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:12.017465 17176 raft_consensus.cc:515] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:12.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:12.019155 17176 leader_election.cc:290] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad837748684c4c93a572185eda10fa14 (127.16.10.194:44603), 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871)
14:00:12.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:12.023002 16728 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.10.195:40871: connect: Connection refused (error 111)
14:00:12.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:12.024652 16728 leader_election.cc:336] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871): Network error: Client connection negotiation failed: client connection to 127.16.10.195:40871: connect: Connection refused (error 111)
14:00:12.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:12.024725 17181 raft_consensus.cc:493] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 948ae7bedcae4f2ea8a1825119425668)
14:00:12.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:12.024950 17181 raft_consensus.cc:515] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:12.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:12.026806 17181 leader_election.cc:290] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871), 52f4ec85820b4a5897f1e992099d8c08 (127.16.10.193:46221)
14:00:12.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:12.028612 16865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.10.195:40871: connect: Connection refused (error 111)
14:00:12.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:12.029354 16930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf8e0efa2228413d8e2421c76198d284" candidate_uuid: "52f4ec85820b4a5897f1e992099d8c08" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ad837748684c4c93a572185eda10fa14" is_pre_election: true
14:00:12.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:12.029937 16930 raft_consensus.cc:2468] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f4ec85820b4a5897f1e992099d8c08 in term 1.
14:00:12.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:12.030719 16729 leader_election.cc:304] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52f4ec85820b4a5897f1e992099d8c08, ad837748684c4c93a572185eda10fa14; no voters: 948ae7bedcae4f2ea8a1825119425668
14:00:12.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:12.031390 17176 raft_consensus.cc:2804] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 1 FOLLOWER]: Leader pre-election won for term 2
14:00:12.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:12.031518 17176 raft_consensus.cc:493] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 948ae7bedcae4f2ea8a1825119425668)
14:00:12.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:12.031625 17176 raft_consensus.cc:3060] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 1 FOLLOWER]: Advancing to term 2
14:00:12.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:12.032419 16865 leader_election.cc:336] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871): Network error: Client connection negotiation failed: client connection to 127.16.10.195:40871: connect: Connection refused (error 111)
14:00:12.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:12.034119 17176 raft_consensus.cc:515] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:12.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:12.034991 17176 leader_election.cc:290] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [CANDIDATE]: Term 2 election: Requested vote from peers ad837748684c4c93a572185eda10fa14 (127.16.10.194:44603), 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871)
14:00:12.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:12.035552 16930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf8e0efa2228413d8e2421c76198d284" candidate_uuid: "52f4ec85820b4a5897f1e992099d8c08" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ad837748684c4c93a572185eda10fa14"
14:00:12.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:12.035764 16930 raft_consensus.cc:3060] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 1 FOLLOWER]: Advancing to term 2
14:00:12.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:12.036554 16794 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf8e0efa2228413d8e2421c76198d284" candidate_uuid: "ad837748684c4c93a572185eda10fa14" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52f4ec85820b4a5897f1e992099d8c08" is_pre_election: true
14:00:12.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:12.036691 16728 leader_election.cc:336] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871): Network error: Client connection negotiation failed: client connection to 127.16.10.195:40871: connect: Connection refused (error 111)
14:00:12.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:12.036859 16794 raft_consensus.cc:2393] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ad837748684c4c93a572185eda10fa14 in current term 2: Already voted for candidate 52f4ec85820b4a5897f1e992099d8c08 in this term.
14:00:12.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:12.037498 16865 leader_election.cc:304] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad837748684c4c93a572185eda10fa14; no voters: 52f4ec85820b4a5897f1e992099d8c08, 948ae7bedcae4f2ea8a1825119425668
14:00:12.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:12.038684 16930 raft_consensus.cc:2468] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f4ec85820b4a5897f1e992099d8c08 in term 2.
14:00:12.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:12.038887 17181 raft_consensus.cc:2749] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
14:00:12.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:12.039140 16729 leader_election.cc:304] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52f4ec85820b4a5897f1e992099d8c08, ad837748684c4c93a572185eda10fa14; no voters: 948ae7bedcae4f2ea8a1825119425668
14:00:12.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:12.039523 17176 raft_consensus.cc:2804] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 2 FOLLOWER]: Leader election won for term 2
14:00:12.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:12.040023 17176 raft_consensus.cc:697] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 2 LEADER]: Becoming Leader. State: Replica: 52f4ec85820b4a5897f1e992099d8c08, State: Running, Role: LEADER
14:00:12.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:12.040542 17176 consensus_queue.cc:237] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:12.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:12.044574 16469 catalog_manager.cc:5696] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 reported cstate change: term changed from 1 to 2, leader changed from 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195) to 52f4ec85820b4a5897f1e992099d8c08 (127.16.10.193). New cstate: current_term: 2 leader_uuid: "52f4ec85820b4a5897f1e992099d8c08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } health_report { overall_health: HEALTHY } } }
14:00:12.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:12.471839 16728 consensus_peers.cc:597] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 -> Peer 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871): Couldn't send request to peer 948ae7bedcae4f2ea8a1825119425668. Status: Network error: Client connection negotiation failed: client connection to 127.16.10.195:40871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:00:12.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:12.558000 16930 raft_consensus.cc:1275] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 2 FOLLOWER]: Refusing update from remote peer 52f4ec85820b4a5897f1e992099d8c08: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
14:00:12.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:12.558919 17176 consensus_queue.cc:1048] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
14:00:14.244 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:14.244 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)[L],948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871),ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)]}]
14:00:14.245 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=14, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=3511), Traces: [0ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [5ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: connection closed, [5ms] delaying RPC due to: Network error: connection closed, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [12ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [13ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [17ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [17ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [28ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [31ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [33ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [36ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [36ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [68ms] refreshing cache from master, [68ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [71ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [73ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [76ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [76ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [168ms] refreshing cache from master, [169ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [172ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [175ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [178ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [178ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [488ms] refreshing cache from master, [488ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [492ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [494ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [497ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [497ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [668ms] refreshing cache from master, [668ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [674ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [677ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [679ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [679ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [3507ms] refreshing cache from master, [3508ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [3511ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@1711764220(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@76ef25a0 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1561206150 after retry RPC@1025970902) -> (continuation of Deferred@1978772652 after retry RPC@976911949) -> (continuation of Deferred@2026605856 after retry RPC@1261405468) -> (continuation of Deferred@1111868658 after retry RPC@1358276775) -> (continuation of Deferred@692357164 after retry RPC@1409920307) -> (continuation of Deferred@1684809541 after retry RPC@2106949326), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6f803814 -> wakeup thread Time-limited test -> (continuation of Deferred@1561206150 after retry RPC@1025970902) -> (continuation of Deferred@1978772652 after retry RPC@976911949) -> (continuation of Deferred@2026605856 after retry RPC@1261405468) -> (continuation of Deferred@1111868658 after retry RPC@1358276775) -> (continuation of Deferred@692357164 after retry RPC@1409920307) -> (continuation of Deferred@1684809541 after retry RPC@2106949326))) after lookup
14:00:14.246 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] connecting to peer
14:00:14.247 [DEBUG - kudu-nio-2] (Connection.java:757) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] Successfully connected to peer
14:00:14.274 [DEBUG - kudu-nio-2] (Negotiator.java:919) Authenticated connection [id: 0x90b9e58b, L:/127.0.0.1:59040 - R:/127.16.10.193:46221] using TOKEN/null
14:00:14.304 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.16.10.195:40871
14:00:14.306 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

14:00:14.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.307983 16427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu
14:00:14.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu
14:00:14.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/wal
14:00:14.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/data
14:00:14.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:00:14.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:00:14.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:00:14.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:00:14.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:00:14.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:00:14.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/logs
14:00:14.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/data/info.pb
14:00:14.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:00:14.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:00:14.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:00:14.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:00:14.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:00:14.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:00:14.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:00:14.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:00:14.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:00:14.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:00:14.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.10.195:40871
14:00:14.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.10.195
14:00:14.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.10.254:37885,127.16.10.253:33517,127.16.10.252:36395
14:00:14.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45985
14:00:14.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.10.195
14:00:14.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.10.212:33969
14:00:14.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:00:14.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:00:14.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:00:14.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:14.695356 17217 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:00:14.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:14.695775 17217 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:00:14.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:14.695860 17217 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:00:14.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:14.711115 17217 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:00:14.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:14.711445 17217 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.10.195
14:00:14.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.723671 17217 tablet_server_runner.cc:78] Tablet server non-default flags:
14:00:14.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:00:14.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.10.212:33969
14:00:14.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:00:14.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:00:14.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/data
14:00:14.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/wal
14:00:14.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:00:14.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:00:14.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.10.195:40871
14:00:14.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:00:14.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:00:14.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:00:14.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/data/info.pb
14:00:14.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.10.195
14:00:14.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45985
14:00:14.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.10.254:37885,127.16.10.253:33517,127.16.10.252:36395
14:00:14.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:00:14.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:00:14.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:00:14.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:00:14.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:00:14.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.10.195
14:00:14.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/logs
14:00:14.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:00:14.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:00:14.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:00:14.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:00:14.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:00:14.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
14:00:14.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:00:14.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 13:43:23 UTC on bdcb31816ec0
14:00:14.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11958
14:00:14.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
14:00:14.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.725617 17217 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:00:14.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.728272 17217 file_cache.cc:492] Constructed file cache file cache with capacity 419430
14:00:14.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:14.741719 17225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:00:14.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:14.741818 17223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:00:14.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:14.741755 17222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:00:14.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.743273 17217 server_base.cc:1061] running on GCE node
14:00:14.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.744071 17217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:00:14.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.745285 17217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:00:14.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.746626 17217 hybrid_clock.cc:648] HybridClock initialized: now 1779112814746539 us; error 67 us; skew 500 ppm
14:00:14.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.747170 17217 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:00:14.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.750119 17217 webserver.cc:533] Webserver started at http://127.16.10.195:45985/ using document root <none> and password file <none>
14:00:14.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.751135 17217 fs_manager.cc:362] Metadata directory not provided
14:00:14.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.751420 17217 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:00:14.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.758086 17217 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.000s
14:00:14.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.762102 17231 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:00:14.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.764110 17217 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.000s	sys 0.003s
14:00:14.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.764318 17217 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/data,/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/wal
14:00:14.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "948ae7bedcae4f2ea8a1825119425668"
14:00:14.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:00:08 on dist-test-slave-rq0s"
14:00:14.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.765326 17217 fs_report.cc:389] FS layout report
14:00:14.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:00:14.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/wal
14:00:14.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/wal
14:00:14.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/data/data
14:00:14.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:00:14.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:00:14.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:00:14.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:00:14.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:00:14.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:00:14.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:00:14.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:00:14.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:00:14.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:00:14.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:00:14.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.793100 17217 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:00:14.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.794018 17217 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:00:14.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.794286 17217 kserver.cc:163] Server-wide thread pool size limit: 3276
14:00:14.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.795372 17217 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:00:14.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.797693 17238 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
14:00:14.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.804271 17217 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:00:14.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.804409 17217 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.007s	user 0.001s	sys 0.000s
14:00:14.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.804548 17217 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:00:14.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.807947 17217 ts_tablet_manager.cc:616] Registered 1 tablets
14:00:14.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.808104 17217 ts_tablet_manager.cc:595] Time spent register tablets: real 0.004s	user 0.003s	sys 0.000s
14:00:14.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.808645 17238 tablet_bootstrap.cc:492] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668: Bootstrap starting.
14:00:14.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.836028 17238 log.cc:826] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668: Log is configured to *not* fsync() on all Append() calls
14:00:14.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.858001 17238 tablet_bootstrap.cc:492] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:00:14.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.859009 17238 tablet_bootstrap.cc:492] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668: Bootstrap complete.
14:00:14.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.861209 17238 ts_tablet_manager.cc:1403] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668: Time spent bootstrapping tablet: real 0.053s	user 0.029s	sys 0.021s
14:00:14.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.861212 17217 rpc_server.cc:307] RPC server started. Bound to: 127.16.10.195:40871
14:00:14.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.861310 17345 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.10.195:40871 every 8 connection(s)
14:00:14.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.863160 17217 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/data/info.pb
14:00:14.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.869596 17238 raft_consensus.cc:359] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:14.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.871073 17238 raft_consensus.cc:740] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948ae7bedcae4f2ea8a1825119425668, State: Initialized, Role: FOLLOWER
14:00:14.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.872548 16427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu as pid 17217
14:00:14.873 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:00:14.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.874401 17238 consensus_queue.cc:260] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:14.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.884055 17238 ts_tablet_manager.cc:1434] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668: Time spent starting tablet: real 0.023s	user 0.015s	sys 0.003s
14:00:14.893 [DEBUG - kudu-nio-2] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 6 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244, tablet=cf8e0efa2228413d8e2421c76198d284, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:00:14.894 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.16.10.193:46221
14:00:14.895 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

14:00:14.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.897336 16427 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu pid 16696
14:00:14.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.923952 17348 heartbeater.cc:344] Connected to a master server at 127.16.10.253:33517
14:00:14.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.924408 17348 heartbeater.cc:461] Registering TS with master...
14:00:14.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.925519 17348 heartbeater.cc:507] Master 127.16.10.253:33517 requested a full tablet report, sending...
14:00:14.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.925909 16834 generic_service.cc:196] Checking for leaks (request via RPC)
14:00:14.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.926163 17347 heartbeater.cc:344] Connected to a master server at 127.16.10.254:37885
14:00:14.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.926330 17347 heartbeater.cc:461] Registering TS with master...
14:00:14.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.927071 17347 heartbeater.cc:507] Master 127.16.10.254:37885 requested a full tablet report, sending...
14:00:14.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.927487 17346 heartbeater.cc:344] Connected to a master server at 127.16.10.252:36395
14:00:14.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.927623 17346 heartbeater.cc:461] Registering TS with master...
14:00:14.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.928198 17346 heartbeater.cc:507] Master 127.16.10.252:36395 requested a full tablet report, sending...
14:00:14.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.929555 16540 ts_manager.cc:194] Re-registered known tserver with Master: 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871)
14:00:14.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.929634 16469 ts_manager.cc:194] Re-registered known tserver with Master: 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871)
14:00:14.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.930039 16610 ts_manager.cc:194] Re-registered known tserver with Master: 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871)
14:00:14.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.931290 16469 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.16.10.195:52933
14:00:14.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:14.934283 17347 heartbeater.cc:499] Master 127.16.10.254:37885 was elected leader, sending a full tablet report...
14:00:15.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:15.033985 16427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu with pid 16696
14:00:15.057 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:00:15.057 [DEBUG - kudu-nio-2] (Connection.java:239) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] handling channelInactive
14:00:15.057 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] cleaning up while in state READY due to: connection closed
14:00:15.058 [DEBUG - Time-limited test] (Connection.java:748) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] connecting to peer
14:00:15.060 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:15.060 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] cleaning up while in state CONNECTING due to: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:15.060 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:15.085 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:15.085 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)[L],948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871),ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)]}]
14:00:15.086 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=2, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=28), Traces: [0ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [2ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [2ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [23ms] refreshing cache from master, [23ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [27ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@263210758(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@643c1274 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@28daf7de -> wakeup thread Time-limited test)) after lookup
14:00:15.086 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] connecting to peer
14:00:15.088 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:15.088 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] cleaning up while in state CONNECTING due to: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:15.088 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:15.105 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:15.106 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)[L],948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871),ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)]}]
14:00:15.107 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=4, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=49), Traces: [0ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [2ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [2ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [23ms] refreshing cache from master, [23ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [27ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [28ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [30ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [30ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [47ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@263210758(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@643c1274 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1640666378 after retry RPC@508689130), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@28daf7de -> wakeup thread Time-limited test -> (continuation of Deferred@1640666378 after retry RPC@508689130))) after lookup
14:00:15.108 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] connecting to peer
14:00:15.109 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:15.109 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] cleaning up while in state CONNECTING due to: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:15.110 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:15.145 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:15.145 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)[L],948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871),ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)]}]
14:00:15.145 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=6, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=88), Traces: [0ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [2ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [2ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [23ms] refreshing cache from master, [23ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [27ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [28ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [30ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [30ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [47ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [49ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [51ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [52ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [83ms] refreshing cache from master, [83ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [86ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@263210758(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@643c1274 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1640666378 after retry RPC@508689130) -> (continuation of Deferred@162684861 after retry RPC@405107403), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@28daf7de -> wakeup thread Time-limited test -> (continuation of Deferred@1640666378 after retry RPC@508689130) -> (continuation of Deferred@162684861 after retry RPC@405107403))) after lookup
14:00:15.146 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] connecting to peer
14:00:15.148 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:15.148 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] cleaning up while in state CONNECTING due to: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:15.148 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:15.205 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:15.205 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)[L],948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871),ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)]}]
14:00:15.206 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=8, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=148), Traces: [0ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [2ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [2ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [23ms] refreshing cache from master, [23ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [27ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [28ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [30ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [30ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [47ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [49ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [51ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [52ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [83ms] refreshing cache from master, [83ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [86ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [88ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [90ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [90ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [143ms] refreshing cache from master, [143ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [147ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@263210758(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@643c1274 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1640666378 after retry RPC@508689130) -> (continuation of Deferred@162684861 after retry RPC@405107403) -> (continuation of Deferred@800806058 after retry RPC@1489905818), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@28daf7de -> wakeup thread Time-limited test -> (continuation of Deferred@1640666378 after retry RPC@508689130) -> (continuation of Deferred@162684861 after retry RPC@405107403) -> (continuation of Deferred@800806058 after retry RPC@1489905818))) after lookup
14:00:15.207 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] connecting to peer
14:00:15.209 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:15.209 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] cleaning up while in state CONNECTING due to: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:15.210 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:15.665 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:15.665 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)[L],948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871),ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)]}]
14:00:15.666 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=10, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=608), Traces: [0ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [2ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [2ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [23ms] refreshing cache from master, [23ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [27ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [28ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [30ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [30ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [47ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [49ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [51ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [52ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [83ms] refreshing cache from master, [83ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [86ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [88ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [90ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [90ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [143ms] refreshing cache from master, [143ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [147ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [149ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [151ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [152ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [603ms] refreshing cache from master, [603ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [607ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@263210758(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@643c1274 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1640666378 after retry RPC@508689130) -> (continuation of Deferred@162684861 after retry RPC@405107403) -> (continuation of Deferred@800806058 after retry RPC@1489905818) -> (continuation of Deferred@1151549399 after retry RPC@1031046229), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@28daf7de -> wakeup thread Time-limited test -> (continuation of Deferred@1640666378 after retry RPC@508689130) -> (continuation of Deferred@162684861 after retry RPC@405107403) -> (continuation of Deferred@800806058 after retry RPC@1489905818) -> (continuation of Deferred@1151549399 after retry RPC@1031046229))) after lookup
14:00:15.667 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] connecting to peer
14:00:15.668 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:15.668 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] cleaning up while in state CONNECTING due to: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:15.668 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:16.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:16.426831 17363 raft_consensus.cc:493] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 52f4ec85820b4a5897f1e992099d8c08)
14:00:16.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:16.427016 17363 raft_consensus.cc:515] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:16.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:16.427583 17363 leader_election.cc:290] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871), 52f4ec85820b4a5897f1e992099d8c08 (127.16.10.193:46221)
14:00:16.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:16.430843 16865 leader_election.cc:336] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 52f4ec85820b4a5897f1e992099d8c08 (127.16.10.193:46221): Network error: Client connection negotiation failed: client connection to 127.16.10.193:46221: connect: Connection refused (error 111)
14:00:16.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:16.437449 17300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf8e0efa2228413d8e2421c76198d284" candidate_uuid: "ad837748684c4c93a572185eda10fa14" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "948ae7bedcae4f2ea8a1825119425668" is_pre_election: true
14:00:16.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:16.438071 17300 raft_consensus.cc:2468] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad837748684c4c93a572185eda10fa14 in term 1.
14:00:16.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:16.439390 16865 leader_election.cc:304] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 948ae7bedcae4f2ea8a1825119425668, ad837748684c4c93a572185eda10fa14; no voters: 52f4ec85820b4a5897f1e992099d8c08
14:00:16.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:16.439790 17363 raft_consensus.cc:2804] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 2 FOLLOWER]: Leader pre-election won for term 3
14:00:16.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:16.439916 17363 raft_consensus.cc:493] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 52f4ec85820b4a5897f1e992099d8c08)
14:00:16.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:16.439982 17363 raft_consensus.cc:3060] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 2 FOLLOWER]: Advancing to term 3
14:00:16.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:16.441963 17363 raft_consensus.cc:515] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:16.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:16.442677 17363 leader_election.cc:290] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [CANDIDATE]: Term 3 election: Requested vote from peers 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871), 52f4ec85820b4a5897f1e992099d8c08 (127.16.10.193:46221)
14:00:16.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:16.443200 17300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf8e0efa2228413d8e2421c76198d284" candidate_uuid: "ad837748684c4c93a572185eda10fa14" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "948ae7bedcae4f2ea8a1825119425668"
14:00:16.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:16.443427 17300 raft_consensus.cc:3060] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 1 FOLLOWER]: Advancing to term 3
14:00:16.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:16.444377 16865 leader_election.cc:336] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 52f4ec85820b4a5897f1e992099d8c08 (127.16.10.193:46221): Network error: Client connection negotiation failed: client connection to 127.16.10.193:46221: connect: Connection refused (error 111)
14:00:16.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:16.447326 17300 raft_consensus.cc:2468] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad837748684c4c93a572185eda10fa14 in term 3.
14:00:16.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:16.447849 16865 leader_election.cc:304] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 948ae7bedcae4f2ea8a1825119425668, ad837748684c4c93a572185eda10fa14; no voters: 52f4ec85820b4a5897f1e992099d8c08
14:00:16.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:16.448249 17363 raft_consensus.cc:2804] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 3 FOLLOWER]: Leader election won for term 3
14:00:16.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:16.448755 17363 raft_consensus.cc:697] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 3 LEADER]: Becoming Leader. State: Replica: ad837748684c4c93a572185eda10fa14, State: Running, Role: LEADER
14:00:16.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:16.449307 17363 consensus_queue.cc:237] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:16.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:16.453567 16469 catalog_manager.cc:5696] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 reported cstate change: term changed from 2 to 3, leader changed from 52f4ec85820b4a5897f1e992099d8c08 (127.16.10.193) to ad837748684c4c93a572185eda10fa14 (127.16.10.194). New cstate: current_term: 3 leader_uuid: "ad837748684c4c93a572185eda10fa14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } health_report { overall_health: UNKNOWN } } }
14:00:16.765 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:16.765 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221),948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871),ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)[L]]}]
14:00:16.766 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=12, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=1708), Traces: [0ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [2ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [2ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [23ms] refreshing cache from master, [23ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [27ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [28ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [30ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [30ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [43ms] refreshing cache from master, [43ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [47ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [49ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [51ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [52ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [83ms] refreshing cache from master, [83ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [86ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [88ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [90ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [90ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [143ms] refreshing cache from master, [143ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [147ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [149ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [151ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [152ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [603ms] refreshing cache from master, [603ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [607ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [608ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [610ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [610ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [1703ms] refreshing cache from master, [1703ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [1706ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@263210758(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@643c1274 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1640666378 after retry RPC@508689130) -> (continuation of Deferred@162684861 after retry RPC@405107403) -> (continuation of Deferred@800806058 after retry RPC@1489905818) -> (continuation of Deferred@1151549399 after retry RPC@1031046229) -> (continuation of Deferred@1382840011 after retry RPC@1260586751), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@28daf7de -> wakeup thread Time-limited test -> (continuation of Deferred@1640666378 after retry RPC@508689130) -> (continuation of Deferred@162684861 after retry RPC@405107403) -> (continuation of Deferred@800806058 after retry RPC@1489905818) -> (continuation of Deferred@1151549399 after retry RPC@1031046229) -> (continuation of Deferred@1382840011 after retry RPC@1260586751))) after lookup
14:00:16.767 [DEBUG - kudu-nio-2] (Connection.java:748) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] connecting to peer
14:00:16.768 [DEBUG - kudu-nio-1] (Connection.java:757) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] Successfully connected to peer
14:00:16.795 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0xecb3819b, L:/127.0.0.1:56798 - R:/127.16.10.194:44603] using TOKEN/null
14:00:16.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:16.802131 17300 raft_consensus.cc:1275] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 3 FOLLOWER]: Refusing update from remote peer ad837748684c4c93a572185eda10fa14: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
14:00:16.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:16.802670 16865 consensus_peers.cc:597] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 -> Peer 52f4ec85820b4a5897f1e992099d8c08 (127.16.10.193:46221): Couldn't send request to peer 52f4ec85820b4a5897f1e992099d8c08. Status: Network error: Client connection negotiation failed: client connection to 127.16.10.193:46221: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:00:16.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:16.803108 17368 consensus_queue.cc:1048] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 4, Time since last communication: 0.000s
14:00:16.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:16.815923 17384 mvcc.cc:204] Tried to move back new op lower bound from 7287246097611091968 to 7287246096179900416. Current Snapshot: MvccSnapshot[applied={T|T < 7287246097611091968}]
14:00:16.842 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.16.10.193:46221
14:00:16.843 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

14:00:16.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:16.844004 16427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu
14:00:16.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu
14:00:16.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/wal
14:00:16.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/data
14:00:16.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:00:16.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:00:16.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:00:16.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:00:16.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:00:16.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:00:16.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/logs
14:00:16.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/data/info.pb
14:00:16.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:00:16.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:00:16.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:00:16.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:00:16.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:00:16.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:00:16.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:00:16.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:00:16.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:00:16.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:00:16.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.10.193:46221
14:00:16.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.10.193
14:00:16.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.10.254:37885,127.16.10.253:33517,127.16.10.252:36395
14:00:16.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44113
14:00:16.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.10.193
14:00:16.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.10.212:33969
14:00:16.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:00:16.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:00:16.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:00:17.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:17.210736 17391 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:00:17.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:17.211079 17391 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:00:17.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:17.211164 17391 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:00:17.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:17.221122 17391 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:00:17.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:17.221310 17391 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.10.193
14:00:17.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.231912 17391 tablet_server_runner.cc:78] Tablet server non-default flags:
14:00:17.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:00:17.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.10.212:33969
14:00:17.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:00:17.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:00:17.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/data
14:00:17.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/wal
14:00:17.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:00:17.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:00:17.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.10.193:46221
14:00:17.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:00:17.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:00:17.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:00:17.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/data/info.pb
14:00:17.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.10.193
14:00:17.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44113
14:00:17.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.10.254:37885,127.16.10.253:33517,127.16.10.252:36395
14:00:17.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:00:17.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:00:17.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:00:17.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:00:17.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:00:17.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.10.193
14:00:17.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/logs
14:00:17.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:00:17.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:00:17.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:00:17.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:00:17.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:00:17.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
14:00:17.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:00:17.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 13:43:23 UTC on bdcb31816ec0
14:00:17.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11958
14:00:17.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
14:00:17.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.233871 17391 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:00:17.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.236282 17391 file_cache.cc:492] Constructed file cache file cache with capacity 419430
14:00:17.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:17.247833 17397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:00:17.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:17.247906 17398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:00:17.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:17.248557 17400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:00:17.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.249470 17391 server_base.cc:1061] running on GCE node
14:00:17.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.250083 17391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:00:17.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.251181 17391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:00:17.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.252406 17391 hybrid_clock.cc:648] HybridClock initialized: now 1779112817252348 us; error 53 us; skew 500 ppm
14:00:17.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.252827 17391 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:00:17.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.255384 17391 webserver.cc:533] Webserver started at http://127.16.10.193:44113/ using document root <none> and password file <none>
14:00:17.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.256233 17391 fs_manager.cc:362] Metadata directory not provided
14:00:17.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.256384 17391 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:00:17.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.262315 17391 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.006s	sys 0.000s
14:00:17.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.265632 17406 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:00:17.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.267248 17391 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
14:00:17.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.267467 17391 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/data,/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/wal
14:00:17.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "52f4ec85820b4a5897f1e992099d8c08"
14:00:17.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:00:07 on dist-test-slave-rq0s"
14:00:17.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.268318 17391 fs_report.cc:389] FS layout report
14:00:17.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:00:17.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/wal
14:00:17.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/wal
14:00:17.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/data/data
14:00:17.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:00:17.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:00:17.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:00:17.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:00:17.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:00:17.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:00:17.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:00:17.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:00:17.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:00:17.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:00:17.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:00:17.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.287953 17391 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:00:17.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.289103 17391 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:00:17.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.289434 17391 kserver.cc:163] Server-wide thread pool size limit: 3276
14:00:17.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.290632 17391 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:00:17.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.293511 17413 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
14:00:17.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.300289 17391 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:00:17.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.300422 17391 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.007s	user 0.001s	sys 0.001s
14:00:17.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.300554 17391 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:00:17.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.304952 17391 ts_tablet_manager.cc:616] Registered 1 tablets
14:00:17.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.305095 17391 ts_tablet_manager.cc:595] Time spent register tablets: real 0.005s	user 0.002s	sys 0.000s
14:00:17.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.305649 17413 tablet_bootstrap.cc:492] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08: Bootstrap starting.
14:00:17.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.329025 17413 log.cc:826] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08: Log is configured to *not* fsync() on all Append() calls
14:00:17.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.353004 17391 rpc_server.cc:307] RPC server started. Bound to: 127.16.10.193:46221
14:00:17.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.353029 17521 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.10.193:46221 every 8 connection(s)
14:00:17.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.355134 17391 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/data/info.pb
14:00:17.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.355612 17413 tablet_bootstrap.cc:492] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=6 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:00:17.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.356559 17413 tablet_bootstrap.cc:492] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08: Bootstrap complete.
14:00:17.365 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:00:17.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.358639 17413 ts_tablet_manager.cc:1403] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08: Time spent bootstrapping tablet: real 0.054s	user 0.035s	sys 0.016s
14:00:17.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.364804 16427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu as pid 17391
14:00:17.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.370990 17413 raft_consensus.cc:359] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:17.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.372350 17413 raft_consensus.cc:740] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f4ec85820b4a5897f1e992099d8c08, State: Initialized, Role: FOLLOWER
14:00:17.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.373374 17413 consensus_queue.cc:260] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:17.391 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 9 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244, tablet=cf8e0efa2228413d8e2421c76198d284, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:00:17.392 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.16.10.194:44603
14:00:17.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.399688 17413 ts_tablet_manager.cc:1434] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08: Time spent starting tablet: real 0.039s	user 0.030s	sys 0.004s
14:00:17.401 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

14:00:17.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.401317 16427 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu pid 16847
14:00:17.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.408833 17523 heartbeater.cc:344] Connected to a master server at 127.16.10.254:37885
14:00:17.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.409297 17523 heartbeater.cc:461] Registering TS with master...
14:00:17.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.410589 17523 heartbeater.cc:507] Master 127.16.10.254:37885 requested a full tablet report, sending...
14:00:17.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.414434 16469 ts_manager.cc:194] Re-registered known tserver with Master: 52f4ec85820b4a5897f1e992099d8c08 (127.16.10.193:46221)
14:00:17.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.416821 16469 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.16.10.193:59699
14:00:17.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.418327 17522 heartbeater.cc:344] Connected to a master server at 127.16.10.252:36395
14:00:17.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.418531 17522 heartbeater.cc:461] Registering TS with master...
14:00:17.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.420715 17523 heartbeater.cc:499] Master 127.16.10.254:37885 was elected leader, sending a full tablet report...
14:00:17.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.421336 17522 heartbeater.cc:507] Master 127.16.10.252:36395 requested a full tablet report, sending...
14:00:17.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.423105 16610 ts_manager.cc:194] Re-registered known tserver with Master: 52f4ec85820b4a5897f1e992099d8c08 (127.16.10.193:46221)
14:00:17.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.425623 16971 generic_service.cc:196] Checking for leaks (request via RPC)
14:00:17.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.436144 17524 heartbeater.cc:344] Connected to a master server at 127.16.10.253:33517
14:00:17.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.436383 17524 heartbeater.cc:461] Registering TS with master...
14:00:17.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.436782 17524 heartbeater.cc:507] Master 127.16.10.253:33517 requested a full tablet report, sending...
14:00:17.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.438287 16540 ts_manager.cc:194] Re-registered known tserver with Master: 52f4ec85820b4a5897f1e992099d8c08 (127.16.10.193:46221)
14:00:17.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:17.532814 16427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu with pid 16847
14:00:17.552 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:00:17.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:17.553560 17536 negotiation.cc:336] Failed RPC negotiation. Trace:
14:00:17.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:00:17.412991 (+     0us) reactor.cc:678] Submitting negotiation task for server connection from 127.16.10.194:45251 (local address 127.16.10.193:46221)
14:00:17.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:00:17.415005 (+  2014us) server_negotiation.cc:207] Beginning negotiation
14:00:17.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:00:17.415018 (+    13us) server_negotiation.cc:400] Waiting for connection header
14:00:17.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:00:17.532541 (+117523us) server_negotiation.cc:408] Connection header received
14:00:17.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:00:17.532870 (+   329us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
14:00:17.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:00:17.532879 (+     9us) server_negotiation.cc:462] Received NEGOTIATE request from client
14:00:17.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:00:17.533204 (+   325us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
14:00:17.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:00:17.533411 (+   207us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
14:00:17.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:00:17.552012 (+ 18601us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.10.194:45251: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
14:00:17.554 [DEBUG - kudu-nio-1] (Connection.java:239) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] handling channelInactive
14:00:17.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":1846,"thread_start_us":1697,"threads_started":1}
14:00:17.554 [DEBUG - kudu-nio-1] (Connection.java:709) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] cleaning up while in state READY due to: connection closed
14:00:17.554 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location ad837748684c4c93a572185eda10fa14(127.16.10.194:44603) for tablet cf8e0efa2228413d8e2421c76198d284: connection closed
14:00:17.565 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:17.565 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221),948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871),ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)[L]]}]
14:00:17.565 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=2, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=12), Traces: [0ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [1ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [11ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@89727914(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@561ec900 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@76629722 -> wakeup thread Time-limited test)) after lookup
14:00:17.566 [DEBUG - kudu-nio-2] (Connection.java:748) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] connecting to peer
14:00:17.567 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:17.567 [DEBUG - kudu-nio-2] (Connection.java:709) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] cleaning up while in state CONNECTING due to: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:17.568 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location ad837748684c4c93a572185eda10fa14(127.16.10.194:44603) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:17.584 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:17.585 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221),948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871),ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)[L]]}]
14:00:17.585 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=4, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=32), Traces: [0ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [1ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [11ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [13ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [15ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [15ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [27ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [31ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@89727914(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@561ec900 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1805065407 after retry RPC@1465139060), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@76629722 -> wakeup thread Time-limited test -> (continuation of Deferred@1805065407 after retry RPC@1465139060))) after lookup
14:00:17.586 [DEBUG - kudu-nio-2] (Connection.java:748) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] connecting to peer
14:00:17.587 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:17.587 [DEBUG - kudu-nio-3] (Connection.java:709) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] cleaning up while in state CONNECTING due to: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:17.587 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location ad837748684c4c93a572185eda10fa14(127.16.10.194:44603) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:17.625 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:17.625 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221),948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871),ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)[L]]}]
14:00:17.626 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=6, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=73), Traces: [0ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [1ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [11ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [13ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [15ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [15ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [27ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [31ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [33ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [34ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [34ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [68ms] refreshing cache from master, [68ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [72ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@89727914(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@561ec900 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1805065407 after retry RPC@1465139060) -> (continuation of Deferred@741979282 after retry RPC@1516278270), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@76629722 -> wakeup thread Time-limited test -> (continuation of Deferred@1805065407 after retry RPC@1465139060) -> (continuation of Deferred@741979282 after retry RPC@1516278270))) after lookup
14:00:17.626 [DEBUG - kudu-nio-2] (Connection.java:748) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] connecting to peer
14:00:17.627 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:17.627 [DEBUG - kudu-nio-4] (Connection.java:709) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] cleaning up while in state CONNECTING due to: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:17.628 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location ad837748684c4c93a572185eda10fa14(127.16.10.194:44603) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:17.684 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:17.685 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221),948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871),ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)[L]]}]
14:00:17.685 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=8, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=132), Traces: [0ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [1ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [11ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [13ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [15ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [15ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [27ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [31ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [33ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [34ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [34ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [68ms] refreshing cache from master, [68ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [72ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [73ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [75ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [75ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [128ms] refreshing cache from master, [128ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [131ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@89727914(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@561ec900 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1805065407 after retry RPC@1465139060) -> (continuation of Deferred@741979282 after retry RPC@1516278270) -> (continuation of Deferred@1675382465 after retry RPC@1030605861), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@76629722 -> wakeup thread Time-limited test -> (continuation of Deferred@1805065407 after retry RPC@1465139060) -> (continuation of Deferred@741979282 after retry RPC@1516278270) -> (continuation of Deferred@1675382465 after retry RPC@1030605861))) after lookup
14:00:17.686 [DEBUG - kudu-nio-2] (Connection.java:748) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] connecting to peer
14:00:17.687 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:17.687 [DEBUG - kudu-nio-5] (Connection.java:709) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] cleaning up while in state CONNECTING due to: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:17.687 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location ad837748684c4c93a572185eda10fa14(127.16.10.194:44603) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:18.144 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:18.144 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221),948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871),ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)[L]]}]
14:00:18.144 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=10, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=591), Traces: [0ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [1ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [11ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [13ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [15ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [15ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [27ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [31ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [33ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [34ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [34ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [68ms] refreshing cache from master, [68ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [72ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [73ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [75ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [75ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [128ms] refreshing cache from master, [128ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [131ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [133ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [134ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [134ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [587ms] refreshing cache from master, [588ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [590ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@89727914(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@561ec900 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1805065407 after retry RPC@1465139060) -> (continuation of Deferred@741979282 after retry RPC@1516278270) -> (continuation of Deferred@1675382465 after retry RPC@1030605861) -> (continuation of Deferred@2048260541 after retry RPC@692723262), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@76629722 -> wakeup thread Time-limited test -> (continuation of Deferred@1805065407 after retry RPC@1465139060) -> (continuation of Deferred@741979282 after retry RPC@1516278270) -> (continuation of Deferred@1675382465 after retry RPC@1030605861) -> (continuation of Deferred@2048260541 after retry RPC@692723262))) after lookup
14:00:18.145 [DEBUG - kudu-nio-2] (Connection.java:748) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] connecting to peer
14:00:18.146 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:18.146 [DEBUG - kudu-nio-6] (Connection.java:709) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] cleaning up while in state CONNECTING due to: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:18.146 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location ad837748684c4c93a572185eda10fa14(127.16.10.194:44603) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:18.704 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:18.704 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221),948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871),ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)[L]]}]
14:00:18.705 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=12, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=1152), Traces: [0ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [1ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [11ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [13ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [15ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [15ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [27ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [31ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [33ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [34ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [34ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [68ms] refreshing cache from master, [68ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [72ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [73ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [75ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [75ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [128ms] refreshing cache from master, [128ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [131ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [133ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [134ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [134ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [587ms] refreshing cache from master, [588ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [590ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [592ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [593ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [593ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [1147ms] refreshing cache from master, [1148ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [1151ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@89727914(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@561ec900 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1805065407 after retry RPC@1465139060) -> (continuation of Deferred@741979282 after retry RPC@1516278270) -> (continuation of Deferred@1675382465 after retry RPC@1030605861) -> (continuation of Deferred@2048260541 after retry RPC@692723262) -> (continuation of Deferred@511027958 after retry RPC@1740111080), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@76629722 -> wakeup thread Time-limited test -> (continuation of Deferred@1805065407 after retry RPC@1465139060) -> (continuation of Deferred@741979282 after retry RPC@1516278270) -> (continuation of Deferred@1675382465 after retry RPC@1030605861) -> (continuation of Deferred@2048260541 after retry RPC@692723262) -> (continuation of Deferred@511027958 after retry RPC@1740111080))) after lookup
14:00:18.705 [DEBUG - kudu-nio-2] (Connection.java:748) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] connecting to peer
14:00:18.707 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:18.707 [DEBUG - kudu-nio-7] (Connection.java:709) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] cleaning up while in state CONNECTING due to: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:18.707 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location ad837748684c4c93a572185eda10fa14(127.16.10.194:44603) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:18.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:18.848551 17538 raft_consensus.cc:493] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ad837748684c4c93a572185eda10fa14)
14:00:18.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:18.848759 17538 raft_consensus.cc:515] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:18.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:18.850278 17538 leader_election.cc:290] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ad837748684c4c93a572185eda10fa14 (127.16.10.194:44603), 52f4ec85820b4a5897f1e992099d8c08 (127.16.10.193:46221)
14:00:18.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:18.851413 17234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.10.194:44603: connect: Connection refused (error 111)
14:00:18.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:18.853289 17234 leader_election.cc:336] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ad837748684c4c93a572185eda10fa14 (127.16.10.194:44603): Network error: Client connection negotiation failed: client connection to 127.16.10.194:44603: connect: Connection refused (error 111)
14:00:18.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:18.859601 17476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf8e0efa2228413d8e2421c76198d284" candidate_uuid: "948ae7bedcae4f2ea8a1825119425668" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "52f4ec85820b4a5897f1e992099d8c08" is_pre_election: true
14:00:18.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:18.860059 17476 raft_consensus.cc:2468] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948ae7bedcae4f2ea8a1825119425668 in term 2.
14:00:18.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:18.861447 17233 leader_election.cc:304] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52f4ec85820b4a5897f1e992099d8c08, 948ae7bedcae4f2ea8a1825119425668; no voters: ad837748684c4c93a572185eda10fa14
14:00:18.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:18.861961 17538 raft_consensus.cc:2804] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 3 FOLLOWER]: Leader pre-election won for term 4
14:00:18.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:18.862104 17538 raft_consensus.cc:493] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ad837748684c4c93a572185eda10fa14)
14:00:18.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:18.862190 17538 raft_consensus.cc:3060] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 3 FOLLOWER]: Advancing to term 4
14:00:18.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:18.865111 17538 raft_consensus.cc:515] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:18.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:18.865845 17538 leader_election.cc:290] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [CANDIDATE]: Term 4 election: Requested vote from peers ad837748684c4c93a572185eda10fa14 (127.16.10.194:44603), 52f4ec85820b4a5897f1e992099d8c08 (127.16.10.193:46221)
14:00:18.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:18.866480 17476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf8e0efa2228413d8e2421c76198d284" candidate_uuid: "948ae7bedcae4f2ea8a1825119425668" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "52f4ec85820b4a5897f1e992099d8c08"
14:00:18.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:18.866732 17476 raft_consensus.cc:3060] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 2 FOLLOWER]: Advancing to term 4
14:00:18.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:18.867304 17234 leader_election.cc:336] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer ad837748684c4c93a572185eda10fa14 (127.16.10.194:44603): Network error: Client connection negotiation failed: client connection to 127.16.10.194:44603: connect: Connection refused (error 111)
14:00:18.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:18.870349 17476 raft_consensus.cc:2468] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948ae7bedcae4f2ea8a1825119425668 in term 4.
14:00:18.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:18.870898 17233 leader_election.cc:304] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52f4ec85820b4a5897f1e992099d8c08, 948ae7bedcae4f2ea8a1825119425668; no voters: ad837748684c4c93a572185eda10fa14
14:00:18.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:18.871237 17538 raft_consensus.cc:2804] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 4 FOLLOWER]: Leader election won for term 4
14:00:18.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:18.871702 17538 raft_consensus.cc:697] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 4 LEADER]: Becoming Leader. State: Replica: 948ae7bedcae4f2ea8a1825119425668, State: Running, Role: LEADER
14:00:18.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:18.872296 17538 consensus_queue.cc:237] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:18.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:18.876552 16469 catalog_manager.cc:5696] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 reported cstate change: term changed from 3 to 4, leader changed from ad837748684c4c93a572185eda10fa14 (127.16.10.194) to 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195). New cstate: current_term: 4 leader_uuid: "948ae7bedcae4f2ea8a1825119425668" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } health_report { overall_health: UNKNOWN } } }
14:00:19.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:19.313861 17234 consensus_peers.cc:597] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 -> Peer ad837748684c4c93a572185eda10fa14 (127.16.10.194:44603): Couldn't send request to peer ad837748684c4c93a572185eda10fa14. Status: Network error: Client connection negotiation failed: client connection to 127.16.10.194:44603: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:00:19.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:19.486774 17476 raft_consensus.cc:1275] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 4 FOLLOWER]: Refusing update from remote peer 948ae7bedcae4f2ea8a1825119425668: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 4 index: 13. (index mismatch)
14:00:19.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:19.487771 17538 consensus_queue.cc:1048] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 8, Time since last communication: 0.000s
14:00:21.064 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:21.064 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221),948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)[L],ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)]}]
14:00:21.065 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=14, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=3511), Traces: [0ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [1ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [8ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [11ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [13ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [15ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [15ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [27ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [31ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [33ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [34ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [34ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [68ms] refreshing cache from master, [68ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [72ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [73ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [75ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [75ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [128ms] refreshing cache from master, [128ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [131ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [133ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [134ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [134ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [587ms] refreshing cache from master, [588ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [590ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [592ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [593ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [593ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [1147ms] refreshing cache from master, [1148ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [1151ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [1152ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [1154ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [1154ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [3507ms] refreshing cache from master, [3508ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [3511ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@89727914(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@561ec900 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1805065407 after retry RPC@1465139060) -> (continuation of Deferred@741979282 after retry RPC@1516278270) -> (continuation of Deferred@1675382465 after retry RPC@1030605861) -> (continuation of Deferred@2048260541 after retry RPC@692723262) -> (continuation of Deferred@511027958 after retry RPC@1740111080) -> (continuation of Deferred@1673751852 after retry RPC@994811136), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@76629722 -> wakeup thread Time-limited test -> (continuation of Deferred@1805065407 after retry RPC@1465139060) -> (continuation of Deferred@741979282 after retry RPC@1516278270) -> (continuation of Deferred@1675382465 after retry RPC@1030605861) -> (continuation of Deferred@2048260541 after retry RPC@692723262) -> (continuation of Deferred@511027958 after retry RPC@1740111080) -> (continuation of Deferred@1673751852 after retry RPC@994811136))) after lookup
14:00:21.065 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] connecting to peer
14:00:21.067 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] Successfully connected to peer
14:00:21.090 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x798f6a9c, L:/127.0.0.1:40732 - R:/127.16.10.195:40871] using TOKEN/null
14:00:21.131 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.16.10.194:44603
14:00:21.131 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

14:00:21.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.132460 16427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu
14:00:21.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu
14:00:21.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/wal
14:00:21.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/data
14:00:21.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:00:21.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:00:21.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:00:21.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:00:21.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:00:21.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:00:21.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/logs
14:00:21.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/data/info.pb
14:00:21.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:00:21.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:00:21.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:00:21.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:00:21.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:00:21.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:00:21.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:00:21.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:00:21.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:00:21.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:00:21.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.10.194:44603
14:00:21.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.10.194
14:00:21.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.10.254:37885,127.16.10.253:33517,127.16.10.252:36395
14:00:21.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=33241
14:00:21.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.10.194
14:00:21.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.10.212:33969
14:00:21.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:00:21.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:00:21.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:00:21.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:21.483868 17580 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:00:21.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:21.484273 17580 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:00:21.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:21.484382 17580 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:00:21.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:21.493554 17580 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:00:21.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:21.493822 17580 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.10.194
14:00:21.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.504212 17580 tablet_server_runner.cc:78] Tablet server non-default flags:
14:00:21.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:00:21.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.10.212:33969
14:00:21.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:00:21.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:00:21.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/data
14:00:21.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/wal
14:00:21.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:00:21.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:00:21.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.10.194:44603
14:00:21.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:00:21.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:00:21.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:00:21.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/data/info.pb
14:00:21.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.10.194
14:00:21.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=33241
14:00:21.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.10.254:37885,127.16.10.253:33517,127.16.10.252:36395
14:00:21.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:00:21.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:00:21.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:00:21.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:00:21.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:00:21.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.10.194
14:00:21.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/logs
14:00:21.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:00:21.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:00:21.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:00:21.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:00:21.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:00:21.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
14:00:21.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:00:21.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 13:43:23 UTC on bdcb31816ec0
14:00:21.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11958
14:00:21.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
14:00:21.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.506644 17580 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:00:21.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.508785 17580 file_cache.cc:492] Constructed file cache file cache with capacity 419430
14:00:21.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:21.519625 17586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:00:21.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:21.520253 17585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:00:21.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:21.520697 17588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:00:21.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.521250 17580 server_base.cc:1061] running on GCE node
14:00:21.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.521863 17580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:00:21.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.522912 17580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:00:21.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.524150 17580 hybrid_clock.cc:648] HybridClock initialized: now 1779112821524112 us; error 36 us; skew 500 ppm
14:00:21.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.524551 17580 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:00:21.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.527051 17580 webserver.cc:533] Webserver started at http://127.16.10.194:33241/ using document root <none> and password file <none>
14:00:21.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.527881 17580 fs_manager.cc:362] Metadata directory not provided
14:00:21.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.528029 17580 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:00:21.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.533789 17580 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.000s
14:00:21.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.537075 17594 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:00:21.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.538672 17580 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
14:00:21.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.538874 17580 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/data,/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/wal
14:00:21.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "ad837748684c4c93a572185eda10fa14"
14:00:21.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:00:08 on dist-test-slave-rq0s"
14:00:21.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.539651 17580 fs_report.cc:389] FS layout report
14:00:21.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:00:21.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/wal
14:00:21.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/wal
14:00:21.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/data/data
14:00:21.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:00:21.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:00:21.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:00:21.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:00:21.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:00:21.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:00:21.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:00:21.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:00:21.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:00:21.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:00:21.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:00:21.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.567754 17580 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:00:21.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.568765 17580 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:00:21.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.569137 17580 kserver.cc:163] Server-wide thread pool size limit: 3276
14:00:21.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.570264 17580 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:00:21.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.572716 17601 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
14:00:21.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.578882 17580 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:00:21.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.579051 17580 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.007s	user 0.001s	sys 0.000s
14:00:21.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.579195 17580 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:00:21.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.582449 17580 ts_tablet_manager.cc:616] Registered 1 tablets
14:00:21.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.582677 17580 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.000s	sys 0.004s
14:00:21.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.583149 17601 tablet_bootstrap.cc:492] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14: Bootstrap starting.
14:00:21.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.607404 17601 log.cc:826] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14: Log is configured to *not* fsync() on all Append() calls
14:00:21.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.630777 17601 tablet_bootstrap.cc:492] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=9 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:00:21.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.631573 17601 tablet_bootstrap.cc:492] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14: Bootstrap complete.
14:00:21.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.633018 17601 ts_tablet_manager.cc:1403] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14: Time spent bootstrapping tablet: real 0.050s	user 0.030s	sys 0.017s
14:00:21.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.633273 17580 rpc_server.cc:307] RPC server started. Bound to: 127.16.10.194:44603
14:00:21.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.633359 17708 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.10.194:44603 every 8 connection(s)
14:00:21.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.634915 17580 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/data/info.pb
14:00:21.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.641961 16427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu as pid 17580
14:00:21.642 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:00:21.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.641115 17601 raft_consensus.cc:359] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:21.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.644021 17601 raft_consensus.cc:740] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad837748684c4c93a572185eda10fa14, State: Initialized, Role: FOLLOWER
14:00:21.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.645355 17601 consensus_queue.cc:260] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:21.659 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 12 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244, tablet=cf8e0efa2228413d8e2421c76198d284, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:00:21.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.676066 17601 ts_tablet_manager.cc:1434] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14: Time spent starting tablet: real 0.043s	user 0.013s	sys 0.016s
14:00:21.682 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.16.10.195:40871
14:00:21.683 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

14:00:21.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.683615 16427 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu pid 17217
14:00:21.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.702018 17709 heartbeater.cc:344] Connected to a master server at 127.16.10.252:36395
14:00:21.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.702462 17709 heartbeater.cc:461] Registering TS with master...
14:00:21.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.703661 17709 heartbeater.cc:507] Master 127.16.10.252:36395 requested a full tablet report, sending...
14:00:21.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.706966 16610 ts_manager.cc:194] Re-registered known tserver with Master: ad837748684c4c93a572185eda10fa14 (127.16.10.194:44603)
14:00:21.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.727594 17710 heartbeater.cc:344] Connected to a master server at 127.16.10.254:37885
14:00:21.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.727761 17710 heartbeater.cc:461] Registering TS with master...
14:00:21.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.728313 17710 heartbeater.cc:507] Master 127.16.10.254:37885 requested a full tablet report, sending...
14:00:21.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.729800 16469 ts_manager.cc:194] Re-registered known tserver with Master: ad837748684c4c93a572185eda10fa14 (127.16.10.194:44603)
14:00:21.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.731352 16469 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.16.10.194:57933
14:00:21.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.732141 17341 generic_service.cc:196] Checking for leaks (request via RPC)
14:00:21.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.733131 17717 heartbeater.cc:344] Connected to a master server at 127.16.10.253:33517
14:00:21.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.733312 17717 heartbeater.cc:461] Registering TS with master...
14:00:21.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.733656 17717 heartbeater.cc:507] Master 127.16.10.253:33517 requested a full tablet report, sending...
14:00:21.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.734704 17710 heartbeater.cc:499] Master 127.16.10.254:37885 was elected leader, sending a full tablet report...
14:00:21.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.735191 16540 ts_manager.cc:194] Re-registered known tserver with Master: ad837748684c4c93a572185eda10fa14 (127.16.10.194:44603)
14:00:21.839 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:21.839037 16427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu with pid 17217
14:00:21.859 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] handling channelInactive
14:00:21.859 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:00:21.859 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] cleaning up while in state READY due to: connection closed
14:00:21.860 [DEBUG - Time-limited test] (Connection.java:748) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] connecting to peer
14:00:21.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:21.860962 17724 negotiation.cc:336] Failed RPC negotiation. Trace:
14:00:21.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:00:21.737315 (+     0us) reactor.cc:678] Submitting negotiation task for server connection from 127.16.10.195:48445 (local address 127.16.10.194:44603)
14:00:21.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:00:21.739155 (+  1840us) server_negotiation.cc:207] Beginning negotiation
14:00:21.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:00:21.739167 (+    12us) server_negotiation.cc:400] Waiting for connection header
14:00:21.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:00:21.739213 (+    46us) server_negotiation.cc:408] Connection header received
14:00:21.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:00:21.739354 (+   141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
14:00:21.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:00:21.739378 (+    24us) server_negotiation.cc:462] Received NEGOTIATE request from client
14:00:21.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:00:21.739737 (+   359us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
14:00:21.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:00:21.740203 (+   466us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
14:00:21.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:00:21.839782 (+ 99579us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
14:00:21.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:00:21.841208 (+  1426us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
14:00:21.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:00:21.859398 (+ 18190us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.10.195:48445: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
14:00:21.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":1645,"thread_start_us":1455,"threads_started":1}
14:00:21.861 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:21.862 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] cleaning up while in state CONNECTING due to: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:21.862 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:21.884 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:21.884 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221),948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)[L],ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)]}]
14:00:21.885 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=2, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=25), Traces: [0ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [2ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [2ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [24ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@1077721924(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@27f89dde -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@58387903 -> wakeup thread Time-limited test)) after lookup
14:00:21.885 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] connecting to peer
14:00:21.887 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:21.887 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] cleaning up while in state CONNECTING due to: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:21.887 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:21.904 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:21.904 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221),948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)[L],ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)]}]
14:00:21.905 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=4, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=45), Traces: [0ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [2ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [2ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [24ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [25ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [27ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [27ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [40ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [43ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@1077721924(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@27f89dde -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1895832831 after retry RPC@1931180108), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@58387903 -> wakeup thread Time-limited test -> (continuation of Deferred@1895832831 after retry RPC@1931180108))) after lookup
14:00:21.906 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] connecting to peer
14:00:21.907 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:21.907 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] cleaning up while in state CONNECTING due to: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:21.907 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:21.945 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:21.945 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221),948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)[L],ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)]}]
14:00:21.945 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=6, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=85), Traces: [0ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [2ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [2ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [24ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [25ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [27ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [27ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [40ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [43ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [45ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [47ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [47ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [81ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [85ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@1077721924(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@27f89dde -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1895832831 after retry RPC@1931180108) -> (continuation of Deferred@1165765060 after retry RPC@1838018623), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@58387903 -> wakeup thread Time-limited test -> (continuation of Deferred@1895832831 after retry RPC@1931180108) -> (continuation of Deferred@1165765060 after retry RPC@1838018623))) after lookup
14:00:21.946 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] connecting to peer
14:00:21.947 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:21.947 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] cleaning up while in state CONNECTING due to: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:21.947 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:21.964 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:21.965 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221),948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)[L],ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)]}]
14:00:21.965 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=8, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=105), Traces: [0ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [2ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [2ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [24ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [25ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [27ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [27ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [40ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [43ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [45ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [47ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [47ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [81ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [85ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [86ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [87ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [88ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [101ms] refreshing cache from master, [101ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [104ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@1077721924(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@27f89dde -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1895832831 after retry RPC@1931180108) -> (continuation of Deferred@1165765060 after retry RPC@1838018623) -> (continuation of Deferred@1041444222 after retry RPC@1016934027), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@58387903 -> wakeup thread Time-limited test -> (continuation of Deferred@1895832831 after retry RPC@1931180108) -> (continuation of Deferred@1165765060 after retry RPC@1838018623) -> (continuation of Deferred@1041444222 after retry RPC@1016934027))) after lookup
14:00:21.966 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] connecting to peer
14:00:21.967 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:21.968 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] cleaning up while in state CONNECTING due to: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:21.968 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:22.084 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:22.085 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221),948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)[L],ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)]}]
14:00:22.085 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=10, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=225), Traces: [0ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [2ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [2ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [24ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [25ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [27ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [27ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [40ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [43ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [45ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [47ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [47ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [81ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [85ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [86ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [87ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [88ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [101ms] refreshing cache from master, [101ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [104ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [106ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [108ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [108ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [221ms] refreshing cache from master, [221ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [224ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@1077721924(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@27f89dde -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1895832831 after retry RPC@1931180108) -> (continuation of Deferred@1165765060 after retry RPC@1838018623) -> (continuation of Deferred@1041444222 after retry RPC@1016934027) -> (continuation of Deferred@335514696 after retry RPC@1178800586), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@58387903 -> wakeup thread Time-limited test -> (continuation of Deferred@1895832831 after retry RPC@1931180108) -> (continuation of Deferred@1165765060 after retry RPC@1838018623) -> (continuation of Deferred@1041444222 after retry RPC@1016934027) -> (continuation of Deferred@335514696 after retry RPC@1178800586))) after lookup
14:00:22.086 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] connecting to peer
14:00:22.088 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:22.088 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] cleaning up while in state CONNECTING due to: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:22.088 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:22.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:22.789243 17725 raft_consensus.cc:493] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
14:00:22.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:22.789546 17725 raft_consensus.cc:515] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:22.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:22.791273 17725 leader_election.cc:290] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871), 52f4ec85820b4a5897f1e992099d8c08 (127.16.10.193:46221)
14:00:22.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:22.792922 17596 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.10.195:40871: connect: Connection refused (error 111)
14:00:22.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:22.795084 17596 leader_election.cc:336] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871): Network error: Client connection negotiation failed: client connection to 127.16.10.195:40871: connect: Connection refused (error 111)
14:00:22.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:22.798205 17476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf8e0efa2228413d8e2421c76198d284" candidate_uuid: "ad837748684c4c93a572185eda10fa14" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "52f4ec85820b4a5897f1e992099d8c08" is_pre_election: true
14:00:22.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:22.799080 17596 leader_election.cc:304] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad837748684c4c93a572185eda10fa14; no voters: 52f4ec85820b4a5897f1e992099d8c08, 948ae7bedcae4f2ea8a1825119425668
14:00:22.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:22.799502 17725 raft_consensus.cc:2749] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
14:00:23.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:23.142128 17730 raft_consensus.cc:493] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 948ae7bedcae4f2ea8a1825119425668)
14:00:23.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:23.142360 17730 raft_consensus.cc:515] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:23.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:23.144122 17730 leader_election.cc:290] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ad837748684c4c93a572185eda10fa14 (127.16.10.194:44603), 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871)
14:00:23.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:23.146173 17408 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.10.195:40871: connect: Connection refused (error 111)
14:00:23.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:23.148507 17408 leader_election.cc:336] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871): Network error: Client connection negotiation failed: client connection to 127.16.10.195:40871: connect: Connection refused (error 111)
14:00:23.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:23.154160 17664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf8e0efa2228413d8e2421c76198d284" candidate_uuid: "52f4ec85820b4a5897f1e992099d8c08" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "ad837748684c4c93a572185eda10fa14" is_pre_election: true
14:00:23.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:23.154816 17664 raft_consensus.cc:2468] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f4ec85820b4a5897f1e992099d8c08 in term 3.
14:00:23.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:23.156085 17409 leader_election.cc:304] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52f4ec85820b4a5897f1e992099d8c08, ad837748684c4c93a572185eda10fa14; no voters: 948ae7bedcae4f2ea8a1825119425668
14:00:23.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:23.156610 17730 raft_consensus.cc:2804] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 4 FOLLOWER]: Leader pre-election won for term 5
14:00:23.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:23.156790 17730 raft_consensus.cc:493] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 948ae7bedcae4f2ea8a1825119425668)
14:00:23.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:23.156911 17730 raft_consensus.cc:3060] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 4 FOLLOWER]: Advancing to term 5
14:00:23.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:23.159334 17730 raft_consensus.cc:515] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:23.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:23.160082 17730 leader_election.cc:290] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [CANDIDATE]: Term 5 election: Requested vote from peers ad837748684c4c93a572185eda10fa14 (127.16.10.194:44603), 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871)
14:00:23.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:23.160939 17664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf8e0efa2228413d8e2421c76198d284" candidate_uuid: "52f4ec85820b4a5897f1e992099d8c08" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "ad837748684c4c93a572185eda10fa14"
14:00:23.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:23.161265 17664 raft_consensus.cc:3060] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 3 FOLLOWER]: Advancing to term 5
14:00:23.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:23.161756 17408 leader_election.cc:336] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871): Network error: Client connection negotiation failed: client connection to 127.16.10.195:40871: connect: Connection refused (error 111)
14:00:23.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:23.165307 17664 raft_consensus.cc:2468] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f4ec85820b4a5897f1e992099d8c08 in term 5.
14:00:23.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:23.165875 17409 leader_election.cc:304] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52f4ec85820b4a5897f1e992099d8c08, ad837748684c4c93a572185eda10fa14; no voters: 948ae7bedcae4f2ea8a1825119425668
14:00:23.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:23.166318 17730 raft_consensus.cc:2804] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 5 FOLLOWER]: Leader election won for term 5
14:00:23.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:23.166935 17730 raft_consensus.cc:697] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 5 LEADER]: Becoming Leader. State: Replica: 52f4ec85820b4a5897f1e992099d8c08, State: Running, Role: LEADER
14:00:23.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:23.167733 17730 consensus_queue.cc:237] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 4.16, Last appended by leader: 16, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:23.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:23.171794 16469 catalog_manager.cc:5696] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 reported cstate change: term changed from 4 to 5, leader changed from 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195) to 52f4ec85820b4a5897f1e992099d8c08 (127.16.10.193). New cstate: current_term: 5 leader_uuid: "52f4ec85820b4a5897f1e992099d8c08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } health_report { overall_health: HEALTHY } } }
14:00:23.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:23.597543 17664 raft_consensus.cc:1275] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 5 FOLLOWER]: Refusing update from remote peer 52f4ec85820b4a5897f1e992099d8c08: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 5 index: 17. (index mismatch)
14:00:23.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:23.598683 17730 consensus_queue.cc:1048] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 12, Time since last communication: 0.000s
14:00:23.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:23.613241 17408 consensus_peers.cc:597] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 -> Peer 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871): Couldn't send request to peer 948ae7bedcae4f2ea8a1825119425668. Status: Network error: Client connection negotiation failed: client connection to 127.16.10.195:40871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:00:24.105 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:24.106 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)[L],948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871),ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)]}]
14:00:24.106 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=12, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=2246), Traces: [0ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [2ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [2ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [21ms] refreshing cache from master, [21ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [24ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [25ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [27ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [27ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [40ms] refreshing cache from master, [41ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [43ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [45ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [47ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [47ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [81ms] refreshing cache from master, [81ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [85ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [86ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [87ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [88ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [101ms] refreshing cache from master, [101ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [104ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [106ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [108ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [108ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [221ms] refreshing cache from master, [221ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [224ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [226ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [228ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [228ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [2241ms] refreshing cache from master, [2241ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [2245ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@1077721924(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@27f89dde -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1895832831 after retry RPC@1931180108) -> (continuation of Deferred@1165765060 after retry RPC@1838018623) -> (continuation of Deferred@1041444222 after retry RPC@1016934027) -> (continuation of Deferred@335514696 after retry RPC@1178800586) -> (continuation of Deferred@1525089533 after retry RPC@74744618), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@58387903 -> wakeup thread Time-limited test -> (continuation of Deferred@1895832831 after retry RPC@1931180108) -> (continuation of Deferred@1165765060 after retry RPC@1838018623) -> (continuation of Deferred@1041444222 after retry RPC@1016934027) -> (continuation of Deferred@335514696 after retry RPC@1178800586) -> (continuation of Deferred@1525089533 after retry RPC@74744618))) after lookup
14:00:24.107 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] connecting to peer
14:00:24.109 [DEBUG - kudu-nio-7] (Connection.java:757) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] Successfully connected to peer
14:00:24.148 [DEBUG - kudu-nio-7] (Negotiator.java:919) Authenticated connection [id: 0x36c7e6a1, L:/127.0.0.1:49010 - R:/127.16.10.193:46221] using TOKEN/null
14:00:24.177 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.16.10.195:40871
14:00:24.177 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

14:00:24.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.178364 16427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu
14:00:24.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu
14:00:24.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/wal
14:00:24.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/data
14:00:24.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:00:24.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:00:24.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:00:24.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:00:24.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:00:24.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:00:24.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/logs
14:00:24.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/data/info.pb
14:00:24.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:00:24.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:00:24.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:00:24.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:00:24.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:00:24.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:00:24.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:00:24.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:00:24.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:00:24.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:00:24.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.10.195:40871
14:00:24.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.10.195
14:00:24.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.10.254:37885,127.16.10.253:33517,127.16.10.252:36395
14:00:24.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45985
14:00:24.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.10.195
14:00:24.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.10.212:33969
14:00:24.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:00:24.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:00:24.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:00:24.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:24.523053 17762 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:00:24.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:24.523428 17762 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:00:24.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:24.523535 17762 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:00:24.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:24.532379 17762 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:00:24.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:24.532655 17762 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.10.195
14:00:24.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.543185 17762 tablet_server_runner.cc:78] Tablet server non-default flags:
14:00:24.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:00:24.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.10.212:33969
14:00:24.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:00:24.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:00:24.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/data
14:00:24.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/wal
14:00:24.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:00:24.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:00:24.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.10.195:40871
14:00:24.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:00:24.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:00:24.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:00:24.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/data/info.pb
14:00:24.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.10.195
14:00:24.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45985
14:00:24.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.10.254:37885,127.16.10.253:33517,127.16.10.252:36395
14:00:24.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:00:24.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:00:24.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:00:24.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:00:24.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:00:24.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.10.195
14:00:24.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/logs
14:00:24.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:00:24.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:00:24.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:00:24.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:00:24.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:00:24.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
14:00:24.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:00:24.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 13:43:23 UTC on bdcb31816ec0
14:00:24.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11958
14:00:24.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
14:00:24.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.545372 17762 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:00:24.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.547457 17762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
14:00:24.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:24.558511 17767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:00:24.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:24.558462 17768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:00:24.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:24.559540 17770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:00:24.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.560880 17762 server_base.cc:1061] running on GCE node
14:00:24.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.561620 17762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:00:24.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.562767 17762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:00:24.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.564018 17762 hybrid_clock.cc:648] HybridClock initialized: now 1779112824563940 us; error 69 us; skew 500 ppm
14:00:24.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.564380 17762 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:00:24.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.567096 17762 webserver.cc:533] Webserver started at http://127.16.10.195:45985/ using document root <none> and password file <none>
14:00:24.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.567945 17762 fs_manager.cc:362] Metadata directory not provided
14:00:24.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.568048 17762 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:00:24.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.573761 17762 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.000s
14:00:24.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.576912 17776 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:00:24.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.578425 17762 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.002s
14:00:24.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.578660 17762 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/data,/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/wal
14:00:24.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "948ae7bedcae4f2ea8a1825119425668"
14:00:24.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:00:08 on dist-test-slave-rq0s"
14:00:24.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.579425 17762 fs_report.cc:389] FS layout report
14:00:24.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:00:24.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/wal
14:00:24.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/wal
14:00:24.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/data/data
14:00:24.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:00:24.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:00:24.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:00:24.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:00:24.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:00:24.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:00:24.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:00:24.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:00:24.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:00:24.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:00:24.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:00:24.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.631376 17762 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:00:24.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.632261 17762 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:00:24.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.632503 17762 kserver.cc:163] Server-wide thread pool size limit: 3276
14:00:24.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.633687 17762 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:00:24.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.636341 17783 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
14:00:24.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.642391 17762 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:00:24.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.642546 17762 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.007s	user 0.001s	sys 0.000s
14:00:24.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.642684 17762 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:00:24.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.646595 17762 ts_tablet_manager.cc:616] Registered 1 tablets
14:00:24.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.646703 17762 ts_tablet_manager.cc:595] Time spent register tablets: real 0.004s	user 0.002s	sys 0.002s
14:00:24.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.647369 17783 tablet_bootstrap.cc:492] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668: Bootstrap starting.
14:00:24.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.675045 17783 log.cc:826] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668: Log is configured to *not* fsync() on all Append() calls
14:00:24.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.695139 17762 rpc_server.cc:307] RPC server started. Bound to: 127.16.10.195:40871
14:00:24.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.695184 17892 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.10.195:40871 every 8 connection(s)
14:00:24.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.696720 17762 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/data/info.pb
14:00:24.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.698676 16427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu as pid 17762
14:00:24.699 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:00:24.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.714390 17783 tablet_bootstrap.cc:492] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=12 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:00:24.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.715026 17783 tablet_bootstrap.cc:492] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668: Bootstrap complete.
14:00:24.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.716313 17783 ts_tablet_manager.cc:1403] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668: Time spent bootstrapping tablet: real 0.070s	user 0.044s	sys 0.009s
14:00:24.726 [DEBUG - kudu-nio-7] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 15 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244, tablet=cf8e0efa2228413d8e2421c76198d284, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:00:24.727 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.16.10.193:46221
14:00:24.727 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

14:00:24.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.728106 16427 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu pid 17391
14:00:24.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.746865 17893 heartbeater.cc:344] Connected to a master server at 127.16.10.252:36395
14:00:24.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.747352 17893 heartbeater.cc:461] Registering TS with master...
14:00:24.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.747881 17894 heartbeater.cc:344] Connected to a master server at 127.16.10.254:37885
14:00:24.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.748039 17894 heartbeater.cc:461] Registering TS with master...
14:00:24.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.748682 17893 heartbeater.cc:507] Master 127.16.10.252:36395 requested a full tablet report, sending...
14:00:24.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.748709 17894 heartbeater.cc:507] Master 127.16.10.254:37885 requested a full tablet report, sending...
14:00:24.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.744831 17783 raft_consensus.cc:359] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:24.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.749198 17783 raft_consensus.cc:740] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948ae7bedcae4f2ea8a1825119425668, State: Initialized, Role: FOLLOWER
14:00:24.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.749992 17783 consensus_queue.cc:260] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 4.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:24.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.751405 17895 heartbeater.cc:344] Connected to a master server at 127.16.10.253:33517
14:00:24.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.751554 17895 heartbeater.cc:461] Registering TS with master...
14:00:24.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.752211 17895 heartbeater.cc:507] Master 127.16.10.253:33517 requested a full tablet report, sending...
14:00:24.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.752398 17783 ts_tablet_manager.cc:1434] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668: Time spent starting tablet: real 0.036s	user 0.006s	sys 0.005s
14:00:24.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.755012 16469 ts_manager.cc:194] Re-registered known tserver with Master: 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871)
14:00:24.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.755123 16610 ts_manager.cc:194] Re-registered known tserver with Master: 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871)
14:00:24.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.755329 16539 ts_manager.cc:194] Re-registered known tserver with Master: 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871)
14:00:24.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.756392 16469 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.16.10.195:33405
14:00:24.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.759687 17894 heartbeater.cc:499] Master 127.16.10.254:37885 was elected leader, sending a full tablet report...
14:00:24.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.760324 17517 generic_service.cc:196] Checking for leaks (request via RPC)
14:00:24.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:24.866999 16427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu with pid 17391
14:00:24.886 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:00:24.886 [DEBUG - kudu-nio-7] (Connection.java:239) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] handling channelInactive
14:00:24.887 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] cleaning up while in state READY due to: connection closed
14:00:24.887 [DEBUG - Time-limited test] (Connection.java:748) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] connecting to peer
14:00:24.888 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:24.889 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] cleaning up while in state CONNECTING due to: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:24.889 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:24.904 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:24.904 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)[L],948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871),ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)]}]
14:00:24.905 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=2, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=18), Traces: [0ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [2ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [2ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [17ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@664274772(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@755ad5e -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@28974053 -> wakeup thread Time-limited test)) after lookup
14:00:24.905 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] connecting to peer
14:00:24.907 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:24.907 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] cleaning up while in state CONNECTING due to: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:24.907 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:24.924 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:24.924 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)[L],948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871),ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)]}]
14:00:24.924 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=4, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=37), Traces: [0ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [2ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [2ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [17ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [18ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [20ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [20ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [36ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@664274772(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@755ad5e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1903605896 after retry RPC@1676802861), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@28974053 -> wakeup thread Time-limited test -> (continuation of Deferred@1903605896 after retry RPC@1676802861))) after lookup
14:00:24.925 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] connecting to peer
14:00:24.926 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:24.926 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] cleaning up while in state CONNECTING due to: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:24.926 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:24.965 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:24.965 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)[L],948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871),ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)]}]
14:00:24.966 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=6, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=78), Traces: [0ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [2ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [2ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [17ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [18ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [20ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [20ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [36ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [38ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [39ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [40ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [74ms] refreshing cache from master, [74ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [77ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@664274772(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@755ad5e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1903605896 after retry RPC@1676802861) -> (continuation of Deferred@1636716771 after retry RPC@230668226), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@28974053 -> wakeup thread Time-limited test -> (continuation of Deferred@1903605896 after retry RPC@1676802861) -> (continuation of Deferred@1636716771 after retry RPC@230668226))) after lookup
14:00:24.967 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] connecting to peer
14:00:24.968 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:24.968 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] cleaning up while in state CONNECTING due to: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:24.969 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:24.984 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:24.985 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)[L],948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871),ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)]}]
14:00:24.985 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=8, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=98), Traces: [0ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [2ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [2ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [17ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [18ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [20ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [20ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [36ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [38ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [39ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [40ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [74ms] refreshing cache from master, [74ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [77ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [79ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [81ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [82ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [94ms] refreshing cache from master, [94ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [97ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@664274772(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@755ad5e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1903605896 after retry RPC@1676802861) -> (continuation of Deferred@1636716771 after retry RPC@230668226) -> (continuation of Deferred@1461568211 after retry RPC@661421653), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@28974053 -> wakeup thread Time-limited test -> (continuation of Deferred@1903605896 after retry RPC@1676802861) -> (continuation of Deferred@1636716771 after retry RPC@230668226) -> (continuation of Deferred@1461568211 after retry RPC@661421653))) after lookup
14:00:24.986 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] connecting to peer
14:00:24.987 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:24.987 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] cleaning up while in state CONNECTING due to: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:24.987 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:25.205 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:25.206 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)[L],948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871),ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)]}]
14:00:25.206 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=10, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=319), Traces: [0ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [2ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [2ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [17ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [18ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [20ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [20ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [36ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [38ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [39ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [40ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [74ms] refreshing cache from master, [74ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [77ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [79ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [81ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [82ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [94ms] refreshing cache from master, [94ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [97ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [99ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [100ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [101ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [314ms] refreshing cache from master, [314ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [318ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@664274772(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@755ad5e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1903605896 after retry RPC@1676802861) -> (continuation of Deferred@1636716771 after retry RPC@230668226) -> (continuation of Deferred@1461568211 after retry RPC@661421653) -> (continuation of Deferred@2056143010 after retry RPC@1054087024), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@28974053 -> wakeup thread Time-limited test -> (continuation of Deferred@1903605896 after retry RPC@1676802861) -> (continuation of Deferred@1636716771 after retry RPC@230668226) -> (continuation of Deferred@1461568211 after retry RPC@661421653) -> (continuation of Deferred@2056143010 after retry RPC@1054087024))) after lookup
14:00:25.207 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] connecting to peer
14:00:25.208 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:25.208 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] cleaning up while in state CONNECTING due to: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:25.208 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:26.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.093246 17909 raft_consensus.cc:493] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 52f4ec85820b4a5897f1e992099d8c08)
14:00:26.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.093603 17909 raft_consensus.cc:515] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:26.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.094519 17909 leader_election.cc:290] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871), 52f4ec85820b4a5897f1e992099d8c08 (127.16.10.193:46221)
14:00:26.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:26.097541 17596 leader_election.cc:336] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 52f4ec85820b4a5897f1e992099d8c08 (127.16.10.193:46221): Network error: Client connection negotiation failed: client connection to 127.16.10.193:46221: connect: Connection refused (error 111)
14:00:26.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.102986 17847 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf8e0efa2228413d8e2421c76198d284" candidate_uuid: "ad837748684c4c93a572185eda10fa14" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "948ae7bedcae4f2ea8a1825119425668" is_pre_election: true
14:00:26.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.103448 17847 raft_consensus.cc:2468] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad837748684c4c93a572185eda10fa14 in term 4.
14:00:26.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.104579 17596 leader_election.cc:304] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 948ae7bedcae4f2ea8a1825119425668, ad837748684c4c93a572185eda10fa14; no voters: 52f4ec85820b4a5897f1e992099d8c08
14:00:26.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.104918 17909 raft_consensus.cc:2804] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 5 FOLLOWER]: Leader pre-election won for term 6
14:00:26.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.105048 17909 raft_consensus.cc:493] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 52f4ec85820b4a5897f1e992099d8c08)
14:00:26.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.105120 17909 raft_consensus.cc:3060] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 5 FOLLOWER]: Advancing to term 6
14:00:26.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.107218 17909 raft_consensus.cc:515] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:26.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.107983 17909 leader_election.cc:290] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [CANDIDATE]: Term 6 election: Requested vote from peers 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871), 52f4ec85820b4a5897f1e992099d8c08 (127.16.10.193:46221)
14:00:26.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.108560 17847 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf8e0efa2228413d8e2421c76198d284" candidate_uuid: "ad837748684c4c93a572185eda10fa14" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "948ae7bedcae4f2ea8a1825119425668"
14:00:26.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.108752 17847 raft_consensus.cc:3060] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 4 FOLLOWER]: Advancing to term 6
14:00:26.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:26.110296 17596 leader_election.cc:336] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 52f4ec85820b4a5897f1e992099d8c08 (127.16.10.193:46221): Network error: Client connection negotiation failed: client connection to 127.16.10.193:46221: connect: Connection refused (error 111)
14:00:26.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.112903 17847 raft_consensus.cc:2468] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad837748684c4c93a572185eda10fa14 in term 6.
14:00:26.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.113490 17596 leader_election.cc:304] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 948ae7bedcae4f2ea8a1825119425668, ad837748684c4c93a572185eda10fa14; no voters: 52f4ec85820b4a5897f1e992099d8c08
14:00:26.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.113855 17909 raft_consensus.cc:2804] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 6 FOLLOWER]: Leader election won for term 6
14:00:26.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.114388 17909 raft_consensus.cc:697] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 6 LEADER]: Becoming Leader. State: Replica: ad837748684c4c93a572185eda10fa14, State: Running, Role: LEADER
14:00:26.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.115038 17909 consensus_queue.cc:237] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 5.20, Last appended by leader: 20, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:26.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.118909 16469 catalog_manager.cc:5696] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 reported cstate change: term changed from 5 to 6, leader changed from 52f4ec85820b4a5897f1e992099d8c08 (127.16.10.193) to ad837748684c4c93a572185eda10fa14 (127.16.10.194). New cstate: current_term: 6 leader_uuid: "ad837748684c4c93a572185eda10fa14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } health_report { overall_health: UNKNOWN } } }
14:00:26.305 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:26.305 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221),948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871),ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)[L]]}]
14:00:26.305 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=12, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=1418), Traces: [0ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [2ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [2ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [17ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [18ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [20ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [20ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [36ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [38ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [39ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [40ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [74ms] refreshing cache from master, [74ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [77ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [79ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [81ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [82ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [94ms] refreshing cache from master, [94ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [97ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [99ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [100ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [101ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [314ms] refreshing cache from master, [314ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [318ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [320ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [321ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [322ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [1414ms] refreshing cache from master, [1414ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [1418ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@664274772(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@755ad5e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1903605896 after retry RPC@1676802861) -> (continuation of Deferred@1636716771 after retry RPC@230668226) -> (continuation of Deferred@1461568211 after retry RPC@661421653) -> (continuation of Deferred@2056143010 after retry RPC@1054087024) -> (continuation of Deferred@1311590758 after retry RPC@1356638276), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@28974053 -> wakeup thread Time-limited test -> (continuation of Deferred@1903605896 after retry RPC@1676802861) -> (continuation of Deferred@1636716771 after retry RPC@230668226) -> (continuation of Deferred@1461568211 after retry RPC@661421653) -> (continuation of Deferred@2056143010 after retry RPC@1054087024) -> (continuation of Deferred@1311590758 after retry RPC@1356638276))) after lookup
14:00:26.307 [DEBUG - kudu-nio-2] (Connection.java:748) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] connecting to peer
14:00:26.308 [DEBUG - kudu-nio-6] (Connection.java:757) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] Successfully connected to peer
14:00:26.336 [DEBUG - kudu-nio-6] (Negotiator.java:919) Authenticated connection [id: 0x19e3709b, L:/127.0.0.1:45798 - R:/127.16.10.194:44603] using TOKEN/null
14:00:26.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.343444 17847 raft_consensus.cc:1275] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 6 FOLLOWER]: Refusing update from remote peer ad837748684c4c93a572185eda10fa14: Log matching property violated. Preceding OpId in replica: term: 4 index: 16. Preceding OpId from leader: term: 6 index: 22. (index mismatch)
14:00:26.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.344405 17914 consensus_queue.cc:1048] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 16, Time since last communication: 0.000s
14:00:26.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:26.345197 17596 consensus_peers.cc:597] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 -> Peer 52f4ec85820b4a5897f1e992099d8c08 (127.16.10.193:46221): Couldn't send request to peer 52f4ec85820b4a5897f1e992099d8c08. Status: Network error: Client connection negotiation failed: client connection to 127.16.10.193:46221: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:00:26.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.368242 17927 mvcc.cc:204] Tried to move back new op lower bound from 7287246136689258496 to 7287246135770607616. Current Snapshot: MvccSnapshot[applied={T|T < 7287246136689258496}]
14:00:26.398 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.16.10.193:46221
14:00:26.399 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

14:00:26.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.400633 16427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu
14:00:26.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu
14:00:26.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/wal
14:00:26.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/data
14:00:26.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:00:26.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:00:26.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:00:26.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:00:26.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:00:26.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:00:26.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/logs
14:00:26.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/data/info.pb
14:00:26.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:00:26.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:00:26.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:00:26.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:00:26.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:00:26.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:00:26.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:00:26.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:00:26.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:00:26.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:00:26.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.10.193:46221
14:00:26.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.10.193
14:00:26.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.10.254:37885,127.16.10.253:33517,127.16.10.252:36395
14:00:26.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44113
14:00:26.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.10.193
14:00:26.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.10.212:33969
14:00:26.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:00:26.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:00:26.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:00:26.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:26.760596 17936 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:00:26.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:26.760978 17936 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:00:26.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:26.761090 17936 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:00:26.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:26.770233 17936 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:00:26.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:26.770476 17936 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.10.193
14:00:26.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.781147 17936 tablet_server_runner.cc:78] Tablet server non-default flags:
14:00:26.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:00:26.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.10.212:33969
14:00:26.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:00:26.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:00:26.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/data
14:00:26.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/wal
14:00:26.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:00:26.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:00:26.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.10.193:46221
14:00:26.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:00:26.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:00:26.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:00:26.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/data/info.pb
14:00:26.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.10.193
14:00:26.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44113
14:00:26.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.10.254:37885,127.16.10.253:33517,127.16.10.252:36395
14:00:26.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:00:26.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:00:26.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:00:26.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:00:26.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:00:26.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.10.193
14:00:26.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/logs
14:00:26.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:00:26.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:00:26.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:00:26.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:00:26.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:00:26.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
14:00:26.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:00:26.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 13:43:23 UTC on bdcb31816ec0
14:00:26.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11958
14:00:26.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
14:00:26.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.783411 17936 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:00:26.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.785532 17936 file_cache.cc:492] Constructed file cache file cache with capacity 419430
14:00:26.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:26.796473 17944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:00:26.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:26.796494 17942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:00:26.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:26.796561 17941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:00:26.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.797425 17936 server_base.cc:1061] running on GCE node
14:00:26.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.798051 17936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:00:26.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.799223 17936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:00:26.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.800524 17936 hybrid_clock.cc:648] HybridClock initialized: now 1779112826800463 us; error 44 us; skew 500 ppm
14:00:26.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.800969 17936 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:00:26.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.803720 17936 webserver.cc:533] Webserver started at http://127.16.10.193:44113/ using document root <none> and password file <none>
14:00:26.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.804682 17936 fs_manager.cc:362] Metadata directory not provided
14:00:26.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.804831 17936 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:00:26.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.811103 17936 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.000s
14:00:26.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.814652 17950 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:00:26.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.816381 17936 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.000s
14:00:26.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.816542 17936 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/data,/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/wal
14:00:26.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "52f4ec85820b4a5897f1e992099d8c08"
14:00:26.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:00:07 on dist-test-slave-rq0s"
14:00:26.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.817297 17936 fs_report.cc:389] FS layout report
14:00:26.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:00:26.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/wal
14:00:26.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/wal
14:00:26.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/data/data
14:00:26.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:00:26.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:00:26.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:00:26.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:00:26.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:00:26.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:00:26.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:00:26.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:00:26.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:00:26.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:00:26.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:00:26.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.832306 17936 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:00:26.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.833348 17936 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:00:26.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.833599 17936 kserver.cc:163] Server-wide thread pool size limit: 3276
14:00:26.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.834615 17936 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:00:26.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.836899 17957 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
14:00:26.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.843422 17936 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:00:26.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.843544 17936 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.007s	user 0.001s	sys 0.000s
14:00:26.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.843649 17936 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:00:26.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.847084 17936 ts_tablet_manager.cc:616] Registered 1 tablets
14:00:26.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.847199 17936 ts_tablet_manager.cc:595] Time spent register tablets: real 0.004s	user 0.000s	sys 0.003s
14:00:26.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.847900 17957 tablet_bootstrap.cc:492] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08: Bootstrap starting.
14:00:26.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.875159 17957 log.cc:826] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08: Log is configured to *not* fsync() on all Append() calls
14:00:26.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.894665 17936 rpc_server.cc:307] RPC server started. Bound to: 127.16.10.193:46221
14:00:26.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.894716 18064 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.10.193:46221 every 8 connection(s)
14:00:26.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.896623 17936 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/data/info.pb
14:00:26.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.901213 16427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu as pid 17936
14:00:26.902 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:00:26.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.910607 17957 tablet_bootstrap.cc:492] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=20 ignored=0} inserts{seen=15 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:00:26.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.911525 17957 tablet_bootstrap.cc:492] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08: Bootstrap complete.
14:00:26.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.913360 17957 ts_tablet_manager.cc:1403] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08: Time spent bootstrapping tablet: real 0.066s	user 0.040s	sys 0.020s
14:00:26.927 [DEBUG - kudu-nio-6] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 18 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244, tablet=cf8e0efa2228413d8e2421c76198d284, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:00:26.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.924002 17957 raft_consensus.cc:359] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:26.928 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.16.10.194:44603
14:00:26.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.927346 17957 raft_consensus.cc:740] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f4ec85820b4a5897f1e992099d8c08, State: Initialized, Role: FOLLOWER
14:00:26.929 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

14:00:26.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.930027 16427 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu pid 17580
14:00:26.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.929666 17957 consensus_queue.cc:260] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 20, Last appended: 5.20, Last appended by leader: 20, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:26.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.936430 18066 heartbeater.cc:344] Connected to a master server at 127.16.10.254:37885
14:00:26.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.936847 18066 heartbeater.cc:461] Registering TS with master...
14:00:26.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.937983 18066 heartbeater.cc:507] Master 127.16.10.254:37885 requested a full tablet report, sending...
14:00:26.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.940155 18067 heartbeater.cc:344] Connected to a master server at 127.16.10.253:33517
14:00:26.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.940318 18067 heartbeater.cc:461] Registering TS with master...
14:00:26.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.940855 18067 heartbeater.cc:507] Master 127.16.10.253:33517 requested a full tablet report, sending...
14:00:26.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.941452 18065 heartbeater.cc:344] Connected to a master server at 127.16.10.252:36395
14:00:26.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.941576 18065 heartbeater.cc:461] Registering TS with master...
14:00:26.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.942255 18065 heartbeater.cc:507] Master 127.16.10.252:36395 requested a full tablet report, sending...
14:00:26.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.943909 16469 ts_manager.cc:194] Re-registered known tserver with Master: 52f4ec85820b4a5897f1e992099d8c08 (127.16.10.193:46221)
14:00:26.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.944408 16610 ts_manager.cc:194] Re-registered known tserver with Master: 52f4ec85820b4a5897f1e992099d8c08 (127.16.10.193:46221)
14:00:26.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.945497 16469 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.16.10.193:39673
14:00:26.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.945510 16539 ts_manager.cc:194] Re-registered known tserver with Master: 52f4ec85820b4a5897f1e992099d8c08 (127.16.10.193:46221)
14:00:26.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.946715 17957 ts_tablet_manager.cc:1434] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08: Time spent starting tablet: real 0.033s	user 0.017s	sys 0.009s
14:00:26.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.950744 18066 heartbeater.cc:499] Master 127.16.10.254:37885 was elected leader, sending a full tablet report...
14:00:26.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:26.951730 17704 generic_service.cc:196] Checking for leaks (request via RPC)
14:00:27.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:27.054653 16427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu with pid 17580
14:00:27.072 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:00:27.073 [DEBUG - kudu-nio-6] (Connection.java:239) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] handling channelInactive
14:00:27.073 [DEBUG - kudu-nio-6] (Connection.java:709) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] cleaning up while in state READY due to: connection closed
14:00:27.082 [DEBUG - pool-2-thread-1] (Connection.java:748) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] connecting to peer
14:00:27.083 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:27.083 [DEBUG - kudu-nio-7] (Connection.java:709) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] cleaning up while in state CONNECTING due to: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:27.083 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location ad837748684c4c93a572185eda10fa14(127.16.10.194:44603) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:27.104 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:27.104 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221),948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871),ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)[L]]}]
14:00:27.104 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=3, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=31), Traces: [0ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [0ms] delaying RPC due to: Illegal state: connection is terminated, [8ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [10ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [10ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [27ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [30ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@1257483367(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@17e9986 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@fe3cb54 -> wakeup thread Time-limited test)) after lookup
14:00:27.106 [DEBUG - kudu-nio-2] (Connection.java:748) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] connecting to peer
14:00:27.107 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:27.107 [DEBUG - kudu-nio-0] (Connection.java:709) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] cleaning up while in state CONNECTING due to: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:27.107 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location ad837748684c4c93a572185eda10fa14(127.16.10.194:44603) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:27.124 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:27.125 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221),948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871),ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)[L]]}]
14:00:27.125 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=5, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=52), Traces: [0ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [0ms] delaying RPC due to: Illegal state: connection is terminated, [8ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [10ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [10ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [27ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [30ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [32ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [34ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [34ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [48ms] refreshing cache from master, [48ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [51ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@1257483367(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@17e9986 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@388729065 after retry RPC@100882483), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@fe3cb54 -> wakeup thread Time-limited test -> (continuation of Deferred@388729065 after retry RPC@100882483))) after lookup
14:00:27.126 [DEBUG - kudu-nio-2] (Connection.java:748) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] connecting to peer
14:00:27.127 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:27.127 [DEBUG - kudu-nio-1] (Connection.java:709) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] cleaning up while in state CONNECTING due to: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:27.127 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location ad837748684c4c93a572185eda10fa14(127.16.10.194:44603) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:27.165 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:27.165 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221),948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871),ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)[L]]}]
14:00:27.165 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=7, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=92), Traces: [0ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [0ms] delaying RPC due to: Illegal state: connection is terminated, [8ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [10ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [10ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [27ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [30ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [32ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [34ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [34ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [48ms] refreshing cache from master, [48ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [51ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [52ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [54ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [54ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [88ms] refreshing cache from master, [88ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [91ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@1257483367(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@17e9986 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@388729065 after retry RPC@100882483) -> (continuation of Deferred@580813820 after retry RPC@1052808540), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@fe3cb54 -> wakeup thread Time-limited test -> (continuation of Deferred@388729065 after retry RPC@100882483) -> (continuation of Deferred@580813820 after retry RPC@1052808540))) after lookup
14:00:27.166 [DEBUG - kudu-nio-2] (Connection.java:748) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] connecting to peer
14:00:27.167 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:27.167 [DEBUG - kudu-nio-2] (Connection.java:709) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] cleaning up while in state CONNECTING due to: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:27.167 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location ad837748684c4c93a572185eda10fa14(127.16.10.194:44603) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:27.405 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:27.405 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221),948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871),ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)[L]]}]
14:00:27.405 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=9, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=332), Traces: [0ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [0ms] delaying RPC due to: Illegal state: connection is terminated, [8ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [10ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [10ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [27ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [30ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [32ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [34ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [34ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [48ms] refreshing cache from master, [48ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [51ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [52ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [54ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [54ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [88ms] refreshing cache from master, [88ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [91ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [93ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [94ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [94ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [328ms] refreshing cache from master, [328ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [331ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@1257483367(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@17e9986 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@388729065 after retry RPC@100882483) -> (continuation of Deferred@580813820 after retry RPC@1052808540) -> (continuation of Deferred@571894942 after retry RPC@908517046), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@fe3cb54 -> wakeup thread Time-limited test -> (continuation of Deferred@388729065 after retry RPC@100882483) -> (continuation of Deferred@580813820 after retry RPC@1052808540) -> (continuation of Deferred@571894942 after retry RPC@908517046))) after lookup
14:00:27.406 [DEBUG - kudu-nio-2] (Connection.java:748) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] connecting to peer
14:00:27.407 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:27.407 [DEBUG - kudu-nio-3] (Connection.java:709) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] cleaning up while in state CONNECTING due to: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:27.408 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location ad837748684c4c93a572185eda10fa14(127.16.10.194:44603) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:28.384 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:28.384 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221),948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871),ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)[L]]}]
14:00:28.385 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=11, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=1311), Traces: [0ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [0ms] delaying RPC due to: Illegal state: connection is terminated, [8ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [10ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [10ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [27ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [30ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [32ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [34ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [34ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [48ms] refreshing cache from master, [48ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [51ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [52ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [54ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [54ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [88ms] refreshing cache from master, [88ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [91ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [93ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [94ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [94ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [328ms] refreshing cache from master, [328ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [331ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [333ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [334ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [335ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [1307ms] refreshing cache from master, [1308ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [1310ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@1257483367(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@17e9986 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@388729065 after retry RPC@100882483) -> (continuation of Deferred@580813820 after retry RPC@1052808540) -> (continuation of Deferred@571894942 after retry RPC@908517046) -> (continuation of Deferred@495646594 after retry RPC@500779499), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@fe3cb54 -> wakeup thread Time-limited test -> (continuation of Deferred@388729065 after retry RPC@100882483) -> (continuation of Deferred@580813820 after retry RPC@1052808540) -> (continuation of Deferred@571894942 after retry RPC@908517046) -> (continuation of Deferred@495646594 after retry RPC@500779499))) after lookup
14:00:28.385 [DEBUG - kudu-nio-2] (Connection.java:748) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] connecting to peer
14:00:28.386 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:28.386 [DEBUG - kudu-nio-4] (Connection.java:709) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] cleaning up while in state CONNECTING due to: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:28.386 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location ad837748684c4c93a572185eda10fa14(127.16.10.194:44603) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:28.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:28.399698 18080 raft_consensus.cc:493] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
14:00:28.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:28.400060 18080 raft_consensus.cc:515] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:28.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:28.401912 18080 leader_election.cc:290] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ad837748684c4c93a572185eda10fa14 (127.16.10.194:44603), 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871)
14:00:28.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:28.403249 17953 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.10.194:44603: connect: Connection refused (error 111)
14:00:28.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:28.405053 17953 leader_election.cc:336] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ad837748684c4c93a572185eda10fa14 (127.16.10.194:44603): Network error: Client connection negotiation failed: client connection to 127.16.10.194:44603: connect: Connection refused (error 111)
14:00:28.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:28.409202 17847 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf8e0efa2228413d8e2421c76198d284" candidate_uuid: "52f4ec85820b4a5897f1e992099d8c08" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "948ae7bedcae4f2ea8a1825119425668" is_pre_election: true
14:00:28.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:28.410153 17952 leader_election.cc:304] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52f4ec85820b4a5897f1e992099d8c08; no voters: 948ae7bedcae4f2ea8a1825119425668, ad837748684c4c93a572185eda10fa14
14:00:28.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:28.410549 18080 raft_consensus.cc:2749] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
14:00:28.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:28.552253 18085 raft_consensus.cc:493] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ad837748684c4c93a572185eda10fa14)
14:00:28.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:28.552467 18085 raft_consensus.cc:515] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:28.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:28.554082 18085 leader_election.cc:290] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ad837748684c4c93a572185eda10fa14 (127.16.10.194:44603), 52f4ec85820b4a5897f1e992099d8c08 (127.16.10.193:46221)
14:00:28.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:28.555442 17779 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.10.194:44603: connect: Connection refused (error 111)
14:00:28.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:28.557215 17779 leader_election.cc:336] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer ad837748684c4c93a572185eda10fa14 (127.16.10.194:44603): Network error: Client connection negotiation failed: client connection to 127.16.10.194:44603: connect: Connection refused (error 111)
14:00:28.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:28.563722 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf8e0efa2228413d8e2421c76198d284" candidate_uuid: "948ae7bedcae4f2ea8a1825119425668" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "52f4ec85820b4a5897f1e992099d8c08" is_pre_election: true
14:00:28.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:28.564224 18019 raft_consensus.cc:2468] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948ae7bedcae4f2ea8a1825119425668 in term 5.
14:00:28.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:28.565394 17778 leader_election.cc:304] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52f4ec85820b4a5897f1e992099d8c08, 948ae7bedcae4f2ea8a1825119425668; no voters: ad837748684c4c93a572185eda10fa14
14:00:28.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:28.565821 18085 raft_consensus.cc:2804] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 6 FOLLOWER]: Leader pre-election won for term 7
14:00:28.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:28.565958 18085 raft_consensus.cc:493] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ad837748684c4c93a572185eda10fa14)
14:00:28.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:28.566085 18085 raft_consensus.cc:3060] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 6 FOLLOWER]: Advancing to term 7
14:00:28.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:28.568229 18085 raft_consensus.cc:515] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:28.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:28.568863 18085 leader_election.cc:290] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [CANDIDATE]: Term 7 election: Requested vote from peers ad837748684c4c93a572185eda10fa14 (127.16.10.194:44603), 52f4ec85820b4a5897f1e992099d8c08 (127.16.10.193:46221)
14:00:28.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:28.569533 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf8e0efa2228413d8e2421c76198d284" candidate_uuid: "948ae7bedcae4f2ea8a1825119425668" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "52f4ec85820b4a5897f1e992099d8c08"
14:00:28.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:28.569758 18019 raft_consensus.cc:3060] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 5 FOLLOWER]: Advancing to term 7
14:00:28.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:28.570541 17779 leader_election.cc:336] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer ad837748684c4c93a572185eda10fa14 (127.16.10.194:44603): Network error: Client connection negotiation failed: client connection to 127.16.10.194:44603: connect: Connection refused (error 111)
14:00:28.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:28.573796 18019 raft_consensus.cc:2468] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948ae7bedcae4f2ea8a1825119425668 in term 7.
14:00:28.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:28.574400 17778 leader_election.cc:304] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52f4ec85820b4a5897f1e992099d8c08, 948ae7bedcae4f2ea8a1825119425668; no voters: ad837748684c4c93a572185eda10fa14
14:00:28.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:28.574812 18085 raft_consensus.cc:2804] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 7 FOLLOWER]: Leader election won for term 7
14:00:28.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:28.575223 18085 raft_consensus.cc:697] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 7 LEADER]: Becoming Leader. State: Replica: 948ae7bedcae4f2ea8a1825119425668, State: Running, Role: LEADER
14:00:28.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:28.575783 18085 consensus_queue.cc:237] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 6.24, Last appended by leader: 24, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:28.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:28.580338 16469 catalog_manager.cc:5696] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 reported cstate change: term changed from 6 to 7, leader changed from ad837748684c4c93a572185eda10fa14 (127.16.10.194) to 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195). New cstate: current_term: 7 leader_uuid: "948ae7bedcae4f2ea8a1825119425668" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } health_report { overall_health: UNKNOWN } } }
14:00:29.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:29.025178 17779 consensus_peers.cc:597] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 -> Peer ad837748684c4c93a572185eda10fa14 (127.16.10.194:44603): Couldn't send request to peer ad837748684c4c93a572185eda10fa14. Status: Network error: Client connection negotiation failed: client connection to 127.16.10.194:44603: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:00:29.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.029663 18019 raft_consensus.cc:1275] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 7 FOLLOWER]: Refusing update from remote peer 948ae7bedcae4f2ea8a1825119425668: Log matching property violated. Preceding OpId in replica: term: 5 index: 20. Preceding OpId from leader: term: 7 index: 25. (index mismatch)
14:00:29.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.030718 18085 consensus_queue.cc:1048] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 20, Time since last communication: 0.000s
14:00:29.124 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:29.125 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221),948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)[L],ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)]}]
14:00:29.125 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=13, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=2052), Traces: [0ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [0ms] delaying RPC due to: Illegal state: connection is terminated, [8ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [10ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [10ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [27ms] refreshing cache from master, [28ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [30ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [32ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [34ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [34ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [48ms] refreshing cache from master, [48ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [51ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [52ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [54ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [54ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [88ms] refreshing cache from master, [88ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [91ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [93ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [94ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [94ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [328ms] refreshing cache from master, [328ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [331ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [333ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [334ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [335ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [1307ms] refreshing cache from master, [1308ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [1310ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [1312ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [1313ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [1314ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [2048ms] refreshing cache from master, [2048ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [2051ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@1257483367(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@17e9986 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@388729065 after retry RPC@100882483) -> (continuation of Deferred@580813820 after retry RPC@1052808540) -> (continuation of Deferred@571894942 after retry RPC@908517046) -> (continuation of Deferred@495646594 after retry RPC@500779499) -> (continuation of Deferred@1751304035 after retry RPC@1067666837), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@fe3cb54 -> wakeup thread Time-limited test -> (continuation of Deferred@388729065 after retry RPC@100882483) -> (continuation of Deferred@580813820 after retry RPC@1052808540) -> (continuation of Deferred@571894942 after retry RPC@908517046) -> (continuation of Deferred@495646594 after retry RPC@500779499) -> (continuation of Deferred@1751304035 after retry RPC@1067666837))) after lookup
14:00:29.126 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] connecting to peer
14:00:29.127 [DEBUG - kudu-nio-5] (Connection.java:757) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] Successfully connected to peer
14:00:29.168 [DEBUG - kudu-nio-5] (Negotiator.java:919) Authenticated connection [id: 0x62be1bdb, L:/127.0.0.1:40784 - R:/127.16.10.195:40871] using TOKEN/null
14:00:29.195 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.16.10.194:44603
14:00:29.196 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

14:00:29.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.196681 16427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu
14:00:29.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu
14:00:29.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/wal
14:00:29.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/data
14:00:29.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:00:29.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:00:29.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:00:29.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:00:29.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:00:29.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:00:29.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/logs
14:00:29.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/data/info.pb
14:00:29.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:00:29.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:00:29.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:00:29.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:00:29.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:00:29.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:00:29.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:00:29.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:00:29.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:00:29.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:00:29.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.10.194:44603
14:00:29.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.10.194
14:00:29.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.10.254:37885,127.16.10.253:33517,127.16.10.252:36395
14:00:29.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=33241
14:00:29.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.10.194
14:00:29.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.10.212:33969
14:00:29.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:00:29.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:00:29.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:00:29.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:29.554178 18112 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:00:29.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:29.554590 18112 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:00:29.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:29.554706 18112 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:00:29.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:29.563665 18112 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:00:29.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:29.563907 18112 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.10.194
14:00:29.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.574434 18112 tablet_server_runner.cc:78] Tablet server non-default flags:
14:00:29.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:00:29.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.10.212:33969
14:00:29.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:00:29.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:00:29.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/data
14:00:29.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/wal
14:00:29.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:00:29.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:00:29.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.10.194:44603
14:00:29.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:00:29.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:00:29.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:00:29.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/data/info.pb
14:00:29.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.10.194
14:00:29.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=33241
14:00:29.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.10.254:37885,127.16.10.253:33517,127.16.10.252:36395
14:00:29.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:00:29.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:00:29.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:00:29.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:00:29.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:00:29.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.10.194
14:00:29.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/logs
14:00:29.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:00:29.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:00:29.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:00:29.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:00:29.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:00:29.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
14:00:29.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:00:29.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 13:43:23 UTC on bdcb31816ec0
14:00:29.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11958
14:00:29.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
14:00:29.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.576821 18112 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:00:29.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.578804 18112 file_cache.cc:492] Constructed file cache file cache with capacity 419430
14:00:29.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:29.588860 18118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:00:29.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:29.588894 18117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:00:29.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:29.590191 18120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:00:29.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.591593 18112 server_base.cc:1061] running on GCE node
14:00:29.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.592208 18112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:00:29.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.593191 18112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:00:29.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.594384 18112 hybrid_clock.cc:648] HybridClock initialized: now 1779112829594326 us; error 59 us; skew 500 ppm
14:00:29.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.594753 18112 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:00:29.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.596925 18112 webserver.cc:533] Webserver started at http://127.16.10.194:33241/ using document root <none> and password file <none>
14:00:29.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.597651 18112 fs_manager.cc:362] Metadata directory not provided
14:00:29.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.597741 18112 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:00:29.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.603587 18112 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.000s
14:00:29.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.606621 18126 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:00:29.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.608217 18112 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
14:00:29.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.608361 18112 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/data,/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/wal
14:00:29.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "ad837748684c4c93a572185eda10fa14"
14:00:29.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:00:08 on dist-test-slave-rq0s"
14:00:29.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.609043 18112 fs_report.cc:389] FS layout report
14:00:29.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:00:29.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/wal
14:00:29.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/wal
14:00:29.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/data/data
14:00:29.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:00:29.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:00:29.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:00:29.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:00:29.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:00:29.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:00:29.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:00:29.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:00:29.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:00:29.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:00:29.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:00:29.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.641482 18112 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:00:29.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.642354 18112 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:00:29.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.642722 18112 kserver.cc:163] Server-wide thread pool size limit: 3276
14:00:29.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.643726 18112 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:00:29.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.646165 18133 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
14:00:29.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.652613 18112 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:00:29.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.652716 18112 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.007s	user 0.001s	sys 0.000s
14:00:29.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.652827 18112 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:00:29.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.655925 18112 ts_tablet_manager.cc:616] Registered 1 tablets
14:00:29.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.656025 18112 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.000s	sys 0.002s
14:00:29.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.656579 18133 tablet_bootstrap.cc:492] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14: Bootstrap starting.
14:00:29.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.682807 18133 log.cc:826] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14: Log is configured to *not* fsync() on all Append() calls
14:00:29.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.711201 18112 rpc_server.cc:307] RPC server started. Bound to: 127.16.10.194:44603
14:00:29.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.711270 18240 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.10.194:44603 every 8 connection(s)
14:00:29.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.713045 18112 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/data/info.pb
14:00:29.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.718600 18133 tablet_bootstrap.cc:492] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14: Bootstrap replayed 1/1 log segments. Stats: ops{read=24 overwritten=0 applied=24 ignored=0} inserts{seen=18 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:00:29.720 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:00:29.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.719493 18133 tablet_bootstrap.cc:492] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14: Bootstrap complete.
14:00:29.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.719595 16427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu as pid 18112
14:00:29.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.721854 18133 ts_tablet_manager.cc:1403] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14: Time spent bootstrapping tablet: real 0.066s	user 0.053s	sys 0.010s
14:00:29.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.729183 18133 raft_consensus.cc:359] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:29.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.730171 18133 raft_consensus.cc:740] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad837748684c4c93a572185eda10fa14, State: Initialized, Role: FOLLOWER
14:00:29.733 [DEBUG - kudu-nio-5] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 21 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244, tablet=cf8e0efa2228413d8e2421c76198d284, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:00:29.734 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.16.10.195:40871
14:00:29.735 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

14:00:29.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.735548 16427 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu pid 17762
14:00:29.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.762161 17888 generic_service.cc:196] Checking for leaks (request via RPC)
14:00:29.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.764030 18133 consensus_queue.cc:260] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 6.24, Last appended by leader: 24, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:29.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.778016 18241 heartbeater.cc:344] Connected to a master server at 127.16.10.252:36395
14:00:29.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.778618 18241 heartbeater.cc:461] Registering TS with master...
14:00:29.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.779037 18242 heartbeater.cc:344] Connected to a master server at 127.16.10.254:37885
14:00:29.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.779166 18242 heartbeater.cc:461] Registering TS with master...
14:00:29.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.780231 18241 heartbeater.cc:507] Master 127.16.10.252:36395 requested a full tablet report, sending...
14:00:29.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.782301 18133 ts_tablet_manager.cc:1434] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14: Time spent starting tablet: real 0.060s	user 0.010s	sys 0.000s
14:00:29.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.782812 18242 heartbeater.cc:507] Master 127.16.10.254:37885 requested a full tablet report, sending...
14:00:29.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.784963 16469 ts_manager.cc:194] Re-registered known tserver with Master: ad837748684c4c93a572185eda10fa14 (127.16.10.194:44603)
14:00:29.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.785346 16610 ts_manager.cc:194] Re-registered known tserver with Master: ad837748684c4c93a572185eda10fa14 (127.16.10.194:44603)
14:00:29.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.786895 16469 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.16.10.194:55883
14:00:29.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.789214 18246 heartbeater.cc:344] Connected to a master server at 127.16.10.253:33517
14:00:29.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.789374 18246 heartbeater.cc:461] Registering TS with master...
14:00:29.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.790123 18246 heartbeater.cc:507] Master 127.16.10.253:33517 requested a full tablet report, sending...
14:00:29.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.790971 18242 heartbeater.cc:499] Master 127.16.10.254:37885 was elected leader, sending a full tablet report...
14:00:29.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.791994 16540 ts_manager.cc:194] Re-registered known tserver with Master: ad837748684c4c93a572185eda10fa14 (127.16.10.194:44603)
14:00:29.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:29.890808 16427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu with pid 17762
14:00:29.913 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:00:29.913 [DEBUG - kudu-nio-5] (Connection.java:239) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] handling channelInactive
14:00:29.914 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] cleaning up while in state READY due to: connection closed
14:00:29.914 [DEBUG - Time-limited test] (Connection.java:748) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] connecting to peer
14:00:29.915 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:29.916 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] cleaning up while in state CONNECTING due to: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:29.916 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:29.924 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:29.924 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221),948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)[L],ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)]}]
14:00:29.925 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=2, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=11), Traces: [0ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [2ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [2ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [10ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@1802166136(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@4e80f865 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@16643830 -> wakeup thread Time-limited test)) after lookup
14:00:29.925 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] connecting to peer
14:00:29.927 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:29.927 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] cleaning up while in state CONNECTING due to: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:29.927 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:29.944 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:29.944 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221),948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)[L],ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)]}]
14:00:29.944 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=4, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=30), Traces: [0ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [2ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [2ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [10ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [11ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [13ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [13ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [29ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@1802166136(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@4e80f865 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@489391557 after retry RPC@716807144), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@16643830 -> wakeup thread Time-limited test -> (continuation of Deferred@489391557 after retry RPC@716807144))) after lookup
14:00:29.945 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] connecting to peer
14:00:29.946 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:29.946 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] cleaning up while in state CONNECTING due to: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:29.946 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:29.984 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:29.984 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221),948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)[L],ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)]}]
14:00:29.985 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=6, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=71), Traces: [0ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [2ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [2ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [10ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [11ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [13ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [13ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [29ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [30ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [32ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [32ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [66ms] refreshing cache from master, [66ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [69ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@1802166136(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@4e80f865 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@489391557 after retry RPC@716807144) -> (continuation of Deferred@840087585 after retry RPC@1418855248), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@16643830 -> wakeup thread Time-limited test -> (continuation of Deferred@489391557 after retry RPC@716807144) -> (continuation of Deferred@840087585 after retry RPC@1418855248))) after lookup
14:00:29.985 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] connecting to peer
14:00:29.986 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:29.987 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] cleaning up while in state CONNECTING due to: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:29.987 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:30.104 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:30.105 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221),948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)[L],ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)]}]
14:00:30.105 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=8, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=191), Traces: [0ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [2ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [2ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [10ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [11ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [13ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [13ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [29ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [30ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [32ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [32ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [66ms] refreshing cache from master, [66ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [69ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [71ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [73ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [73ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [187ms] refreshing cache from master, [187ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [190ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@1802166136(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@4e80f865 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@489391557 after retry RPC@716807144) -> (continuation of Deferred@840087585 after retry RPC@1418855248) -> (continuation of Deferred@2081354361 after retry RPC@952297003), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@16643830 -> wakeup thread Time-limited test -> (continuation of Deferred@489391557 after retry RPC@716807144) -> (continuation of Deferred@840087585 after retry RPC@1418855248) -> (continuation of Deferred@2081354361 after retry RPC@952297003))) after lookup
14:00:30.106 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] connecting to peer
14:00:30.107 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:30.107 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] cleaning up while in state CONNECTING due to: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:30.108 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:30.164 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:30.164 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221),948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)[L],ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)]}]
14:00:30.165 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=10, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=250), Traces: [0ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [2ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [2ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [10ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [11ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [13ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [13ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [29ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [30ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [32ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [32ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [66ms] refreshing cache from master, [66ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [69ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [71ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [73ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [73ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [187ms] refreshing cache from master, [187ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [190ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [192ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [194ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [194ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [247ms] refreshing cache from master, [247ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [250ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@1802166136(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@4e80f865 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@489391557 after retry RPC@716807144) -> (continuation of Deferred@840087585 after retry RPC@1418855248) -> (continuation of Deferred@2081354361 after retry RPC@952297003) -> (continuation of Deferred@795596774 after retry RPC@2106405494), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@16643830 -> wakeup thread Time-limited test -> (continuation of Deferred@489391557 after retry RPC@716807144) -> (continuation of Deferred@840087585 after retry RPC@1418855248) -> (continuation of Deferred@2081354361 after retry RPC@952297003) -> (continuation of Deferred@795596774 after retry RPC@2106405494))) after lookup
14:00:30.165 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] connecting to peer
14:00:30.166 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:30.167 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] cleaning up while in state CONNECTING due to: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:30.167 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:30.305 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:30.305 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221),948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)[L],ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)]}]
14:00:30.305 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=12, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=391), Traces: [0ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [2ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [2ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [10ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [11ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [13ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [13ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [29ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [30ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [32ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [32ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [66ms] refreshing cache from master, [66ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [69ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [71ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [73ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [73ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [187ms] refreshing cache from master, [187ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [190ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [192ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [194ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [194ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [247ms] refreshing cache from master, [247ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [250ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [251ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [253ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [253ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [387ms] refreshing cache from master, [387ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [390ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@1802166136(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@4e80f865 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@489391557 after retry RPC@716807144) -> (continuation of Deferred@840087585 after retry RPC@1418855248) -> (continuation of Deferred@2081354361 after retry RPC@952297003) -> (continuation of Deferred@795596774 after retry RPC@2106405494) -> (continuation of Deferred@1098961140 after retry RPC@1426740272), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@16643830 -> wakeup thread Time-limited test -> (continuation of Deferred@489391557 after retry RPC@716807144) -> (continuation of Deferred@840087585 after retry RPC@1418855248) -> (continuation of Deferred@2081354361 after retry RPC@952297003) -> (continuation of Deferred@795596774 after retry RPC@2106405494) -> (continuation of Deferred@1098961140 after retry RPC@1426740272))) after lookup
14:00:30.306 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] connecting to peer
14:00:30.307 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:30.307 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] cleaning up while in state CONNECTING due to: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:30.307 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:31.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:31.200608 18256 raft_consensus.cc:493] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
14:00:31.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:31.200928 18256 raft_consensus.cc:515] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:31.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:31.202767 18256 leader_election.cc:290] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871), 52f4ec85820b4a5897f1e992099d8c08 (127.16.10.193:46221)
14:00:31.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:31.204736 18128 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.10.195:40871: connect: Connection refused (error 111)
14:00:31.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:31.207053 18128 leader_election.cc:336] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871): Network error: Client connection negotiation failed: client connection to 127.16.10.195:40871: connect: Connection refused (error 111)
14:00:31.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:31.209129 18261 raft_consensus.cc:493] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 948ae7bedcae4f2ea8a1825119425668)
14:00:31.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:31.209674 18261 raft_consensus.cc:515] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:31.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:31.210685 18261 leader_election.cc:290] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ad837748684c4c93a572185eda10fa14 (127.16.10.194:44603), 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871)
14:00:31.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:31.210843 18019 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf8e0efa2228413d8e2421c76198d284" candidate_uuid: "ad837748684c4c93a572185eda10fa14" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "52f4ec85820b4a5897f1e992099d8c08" is_pre_election: true
14:00:31.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:31.211195 18019 raft_consensus.cc:2393] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ad837748684c4c93a572185eda10fa14 in current term 7: Already voted for candidate 948ae7bedcae4f2ea8a1825119425668 in this term.
14:00:31.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:31.212940 18128 leader_election.cc:304] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad837748684c4c93a572185eda10fa14; no voters: 52f4ec85820b4a5897f1e992099d8c08, 948ae7bedcae4f2ea8a1825119425668
14:00:31.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:31.213989 18256 raft_consensus.cc:3060] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 6 FOLLOWER]: Advancing to term 7
14:00:31.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:31.221619 18256 raft_consensus.cc:2749] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
14:00:31.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:31.223097 17952 leader_election.cc:336] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871): Network error: Client connection negotiation failed: client connection to 127.16.10.195:40871: connect: Connection refused (error 111)
14:00:31.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:31.223317 18196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf8e0efa2228413d8e2421c76198d284" candidate_uuid: "52f4ec85820b4a5897f1e992099d8c08" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "ad837748684c4c93a572185eda10fa14" is_pre_election: true
14:00:31.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:31.223726 18196 raft_consensus.cc:2468] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f4ec85820b4a5897f1e992099d8c08 in term 7.
14:00:31.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:31.224808 17953 leader_election.cc:304] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52f4ec85820b4a5897f1e992099d8c08, ad837748684c4c93a572185eda10fa14; no voters: 948ae7bedcae4f2ea8a1825119425668
14:00:31.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:31.225196 18261 raft_consensus.cc:2804] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 7 FOLLOWER]: Leader pre-election won for term 8
14:00:31.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:31.225306 18261 raft_consensus.cc:493] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 948ae7bedcae4f2ea8a1825119425668)
14:00:31.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:31.225374 18261 raft_consensus.cc:3060] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 7 FOLLOWER]: Advancing to term 8
14:00:31.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:31.227566 18261 raft_consensus.cc:515] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:31.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:31.228216 18261 leader_election.cc:290] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [CANDIDATE]: Term 8 election: Requested vote from peers ad837748684c4c93a572185eda10fa14 (127.16.10.194:44603), 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871)
14:00:31.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:31.228715 18196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf8e0efa2228413d8e2421c76198d284" candidate_uuid: "52f4ec85820b4a5897f1e992099d8c08" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "ad837748684c4c93a572185eda10fa14"
14:00:31.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:31.228924 18196 raft_consensus.cc:3060] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 7 FOLLOWER]: Advancing to term 8
14:00:31.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:31.229992 17952 leader_election.cc:336] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871): Network error: Client connection negotiation failed: client connection to 127.16.10.195:40871: connect: Connection refused (error 111)
14:00:31.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:31.231494 18196 raft_consensus.cc:2468] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f4ec85820b4a5897f1e992099d8c08 in term 8.
14:00:31.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:31.231972 17953 leader_election.cc:304] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52f4ec85820b4a5897f1e992099d8c08, ad837748684c4c93a572185eda10fa14; no voters: 948ae7bedcae4f2ea8a1825119425668
14:00:31.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:31.232306 18261 raft_consensus.cc:2804] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 8 FOLLOWER]: Leader election won for term 8
14:00:31.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:31.232985 18261 raft_consensus.cc:697] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 8 LEADER]: Becoming Leader. State: Replica: 52f4ec85820b4a5897f1e992099d8c08, State: Running, Role: LEADER
14:00:31.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:31.234058 18261 consensus_queue.cc:237] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 7.28, Last appended by leader: 28, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:31.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:31.238169 16469 catalog_manager.cc:5696] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 reported cstate change: term changed from 7 to 8, leader changed from 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195) to 52f4ec85820b4a5897f1e992099d8c08 (127.16.10.193). New cstate: current_term: 8 leader_uuid: "52f4ec85820b4a5897f1e992099d8c08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } health_report { overall_health: HEALTHY } } }
14:00:31.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:31.748363 18196 raft_consensus.cc:1275] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 8 FOLLOWER]: Refusing update from remote peer 52f4ec85820b4a5897f1e992099d8c08: Log matching property violated. Preceding OpId in replica: term: 6 index: 24. Preceding OpId from leader: term: 8 index: 29. (index mismatch)
14:00:31.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:31.749401 18279 consensus_queue.cc:1048] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 24, Time since last communication: 0.000s
14:00:31.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:31.779129 17952 consensus_peers.cc:597] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 -> Peer 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871): Couldn't send request to peer 948ae7bedcae4f2ea8a1825119425668. Status: Network error: Client connection negotiation failed: client connection to 127.16.10.195:40871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:00:33.384 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:33.384 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)[L],948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871),ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)]}]
14:00:33.385 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=14, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=3471), Traces: [0ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [2ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [2ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [10ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [11ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [13ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [13ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [29ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [30ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [32ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [32ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [66ms] refreshing cache from master, [66ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [69ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [71ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [73ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [73ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [187ms] refreshing cache from master, [187ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [190ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [192ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [194ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [194ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [247ms] refreshing cache from master, [247ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [250ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [251ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [253ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [253ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [387ms] refreshing cache from master, [387ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [390ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [392ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [393ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [394ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [3467ms] refreshing cache from master, [3467ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [3470ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@1802166136(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@4e80f865 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@489391557 after retry RPC@716807144) -> (continuation of Deferred@840087585 after retry RPC@1418855248) -> (continuation of Deferred@2081354361 after retry RPC@952297003) -> (continuation of Deferred@795596774 after retry RPC@2106405494) -> (continuation of Deferred@1098961140 after retry RPC@1426740272) -> (continuation of Deferred@571331551 after retry RPC@631351418), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@16643830 -> wakeup thread Time-limited test -> (continuation of Deferred@489391557 after retry RPC@716807144) -> (continuation of Deferred@840087585 after retry RPC@1418855248) -> (continuation of Deferred@2081354361 after retry RPC@952297003) -> (continuation of Deferred@795596774 after retry RPC@2106405494) -> (continuation of Deferred@1098961140 after retry RPC@1426740272) -> (continuation of Deferred@571331551 after retry RPC@631351418))) after lookup
14:00:33.386 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] connecting to peer
14:00:33.387 [DEBUG - kudu-nio-5] (Connection.java:757) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] Successfully connected to peer
14:00:33.407 [DEBUG - kudu-nio-5] (Negotiator.java:919) Authenticated connection [id: 0x76ae6739, L:/127.0.0.1:60116 - R:/127.16.10.193:46221] using TOKEN/null
14:00:33.436 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.16.10.195:40871
14:00:33.436 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

14:00:33.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:33.437458 16427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu
14:00:33.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu
14:00:33.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/wal
14:00:33.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/data
14:00:33.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:00:33.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:00:33.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:00:33.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:00:33.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:00:33.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:00:33.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/logs
14:00:33.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/data/info.pb
14:00:33.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:00:33.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:00:33.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:00:33.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:00:33.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:00:33.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:00:33.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:00:33.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:00:33.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:00:33.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:00:33.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.10.195:40871
14:00:33.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.10.195
14:00:33.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.10.254:37885,127.16.10.253:33517,127.16.10.252:36395
14:00:33.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45985
14:00:33.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.10.195
14:00:33.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.10.212:33969
14:00:33.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:00:33.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:00:33.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:00:33.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:33.771170 17952 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.10.195:40871: connect: Connection refused (error 111) [suppressed 6 similar messages]
14:00:33.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:33.790139 18308 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:00:33.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:33.790544 18308 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:00:33.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:33.790644 18308 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:00:33.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:33.799651 18308 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:00:33.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:33.799837 18308 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.10.195
14:00:33.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:33.810683 18308 tablet_server_runner.cc:78] Tablet server non-default flags:
14:00:33.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:00:33.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.10.212:33969
14:00:33.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:00:33.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:00:33.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/data
14:00:33.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/wal
14:00:33.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:00:33.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:00:33.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.10.195:40871
14:00:33.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:00:33.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:00:33.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:00:33.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/data/info.pb
14:00:33.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.10.195
14:00:33.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45985
14:00:33.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.10.254:37885,127.16.10.253:33517,127.16.10.252:36395
14:00:33.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:00:33.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:00:33.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:00:33.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:00:33.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:00:33.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.10.195
14:00:33.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/logs
14:00:33.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:00:33.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:00:33.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:00:33.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:00:33.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:00:33.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
14:00:33.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:00:33.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 13:43:23 UTC on bdcb31816ec0
14:00:33.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11958
14:00:33.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
14:00:33.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:33.812439 18308 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:00:33.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:33.814607 18308 file_cache.cc:492] Constructed file cache file cache with capacity 419430
14:00:33.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:33.831655 18313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:00:33.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:33.831970 18314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:00:33.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:33.833936 18316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:00:33.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:33.834053 18308 server_base.cc:1061] running on GCE node
14:00:33.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:33.834978 18308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:00:33.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:33.836097 18308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:00:33.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:33.837374 18308 hybrid_clock.cc:648] HybridClock initialized: now 1779112833837329 us; error 29 us; skew 500 ppm
14:00:33.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:33.837790 18308 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:00:33.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:33.840672 18308 webserver.cc:533] Webserver started at http://127.16.10.195:45985/ using document root <none> and password file <none>
14:00:33.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:33.841512 18308 fs_manager.cc:362] Metadata directory not provided
14:00:33.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:33.841604 18308 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:00:33.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:33.847304 18308 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.001s	sys 0.003s
14:00:33.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:33.850615 18322 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:00:33.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:33.852329 18308 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.001s
14:00:33.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:33.852474 18308 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/data,/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/wal
14:00:33.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "948ae7bedcae4f2ea8a1825119425668"
14:00:33.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:00:08 on dist-test-slave-rq0s"
14:00:33.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:33.853251 18308 fs_report.cc:389] FS layout report
14:00:33.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:00:33.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/wal
14:00:33.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/wal
14:00:33.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/data/data
14:00:33.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:00:33.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:00:33.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:00:33.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:00:33.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:00:33.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:00:33.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:00:33.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:00:33.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:00:33.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:00:33.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:00:33.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:33.872463 18308 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:00:33.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:33.873363 18308 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:00:33.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:33.873611 18308 kserver.cc:163] Server-wide thread pool size limit: 3276
14:00:33.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:33.874614 18308 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:00:33.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:33.876861 18329 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
14:00:33.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:33.882972 18308 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:00:33.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:33.883126 18308 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.007s	user 0.001s	sys 0.000s
14:00:33.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:33.883268 18308 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:00:33.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:33.886651 18308 ts_tablet_manager.cc:616] Registered 1 tablets
14:00:33.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:33.886782 18308 ts_tablet_manager.cc:595] Time spent register tablets: real 0.004s	user 0.004s	sys 0.000s
14:00:33.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:33.887259 18329 tablet_bootstrap.cc:492] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668: Bootstrap starting.
14:00:33.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:33.915027 18329 log.cc:826] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668: Log is configured to *not* fsync() on all Append() calls
14:00:33.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:33.943960 18308 rpc_server.cc:307] RPC server started. Bound to: 127.16.10.195:40871
14:00:33.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:33.944108 18436 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.10.195:40871 every 8 connection(s)
14:00:33.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:33.945534 18308 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/data/info.pb
14:00:33.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:33.946770 16427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu as pid 18308
14:00:33.948 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:00:33.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:33.966710 18329 tablet_bootstrap.cc:492] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=21 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:00:33.975 [DEBUG - kudu-nio-5] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 24 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244, tablet=cf8e0efa2228413d8e2421c76198d284, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:00:33.976 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.16.10.193:46221
14:00:33.976 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

14:00:33.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:33.977114 16427 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu pid 17936
14:00:33.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:33.995906 18329 tablet_bootstrap.cc:492] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668: Bootstrap complete.
14:00:34.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:34.002806 18329 ts_tablet_manager.cc:1403] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668: Time spent bootstrapping tablet: real 0.116s	user 0.063s	sys 0.009s
14:00:34.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:34.006659 18060 generic_service.cc:196] Checking for leaks (request via RPC)
14:00:34.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:34.007561 18437 heartbeater.cc:344] Connected to a master server at 127.16.10.252:36395
14:00:34.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:34.008025 18437 heartbeater.cc:461] Registering TS with master...
14:00:34.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:34.009351 18437 heartbeater.cc:507] Master 127.16.10.252:36395 requested a full tablet report, sending...
14:00:34.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:34.013886 16610 ts_manager.cc:194] Re-registered known tserver with Master: 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871)
14:00:34.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:34.015071 18329 raft_consensus.cc:359] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:34.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:34.016084 18329 raft_consensus.cc:740] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948ae7bedcae4f2ea8a1825119425668, State: Initialized, Role: FOLLOWER
14:00:34.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:34.016860 18329 consensus_queue.cc:260] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 7.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:34.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:34.029284 18438 heartbeater.cc:344] Connected to a master server at 127.16.10.254:37885
14:00:34.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:34.029513 18438 heartbeater.cc:461] Registering TS with master...
14:00:34.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:34.030239 18438 heartbeater.cc:507] Master 127.16.10.254:37885 requested a full tablet report, sending...
14:00:34.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:34.032006 16469 ts_manager.cc:194] Re-registered known tserver with Master: 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871)
14:00:34.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:34.033753 16469 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.16.10.195:49099
14:00:34.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:34.037918 18329 ts_tablet_manager.cc:1434] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668: Time spent starting tablet: real 0.030s	user 0.012s	sys 0.005s
14:00:34.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:34.040151 18438 heartbeater.cc:499] Master 127.16.10.254:37885 was elected leader, sending a full tablet report...
14:00:34.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:34.050840 18442 heartbeater.cc:344] Connected to a master server at 127.16.10.253:33517
14:00:34.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:34.051028 18442 heartbeater.cc:461] Registering TS with master...
14:00:34.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:34.051369 18442 heartbeater.cc:507] Master 127.16.10.253:33517 requested a full tablet report, sending...
14:00:34.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:34.052901 16540 ts_manager.cc:194] Re-registered known tserver with Master: 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871)
14:00:34.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:34.163210 16427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu with pid 17936
14:00:34.189 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:00:34.189 [DEBUG - kudu-nio-5] (Connection.java:239) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] handling channelInactive
14:00:34.189 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] cleaning up while in state READY due to: connection closed
14:00:34.190 [DEBUG - Time-limited test] (Connection.java:748) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] connecting to peer
14:00:34.191 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:34.191 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] cleaning up while in state CONNECTING due to: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:34.191 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:34.205 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:34.205 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)[L],948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871),ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)]}]
14:00:34.205 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=2, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=15), Traces: [0ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [1ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [1ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [14ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@1503499965(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@7157078c -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@68e624b0 -> wakeup thread Time-limited test)) after lookup
14:00:34.206 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] connecting to peer
14:00:34.207 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:34.207 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] cleaning up while in state CONNECTING due to: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:34.208 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:34.224 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:34.225 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)[L],948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871),ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)]}]
14:00:34.225 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=4, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=35), Traces: [0ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [1ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [1ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [14ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [16ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [18ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [18ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [34ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@1503499965(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@7157078c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@284718322 after retry RPC@1239548409), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@68e624b0 -> wakeup thread Time-limited test -> (continuation of Deferred@284718322 after retry RPC@1239548409))) after lookup
14:00:34.226 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] connecting to peer
14:00:34.227 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:34.228 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] cleaning up while in state CONNECTING due to: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:34.228 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:34.244 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:34.244 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)[L],948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871),ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)]}]
14:00:34.245 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=6, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=55), Traces: [0ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [1ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [1ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [14ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [16ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [18ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [18ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [34ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [36ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [38ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [38ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [50ms] refreshing cache from master, [51ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [53ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@1503499965(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@7157078c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@284718322 after retry RPC@1239548409) -> (continuation of Deferred@1165424210 after retry RPC@482532044), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@68e624b0 -> wakeup thread Time-limited test -> (continuation of Deferred@284718322 after retry RPC@1239548409) -> (continuation of Deferred@1165424210 after retry RPC@482532044))) after lookup
14:00:34.245 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] connecting to peer
14:00:34.246 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:34.247 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] cleaning up while in state CONNECTING due to: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:34.247 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:34.324 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:34.324 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)[L],948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871),ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)]}]
14:00:34.325 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=8, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=135), Traces: [0ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [1ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [1ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [14ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [16ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [18ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [18ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [34ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [36ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [38ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [38ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [50ms] refreshing cache from master, [51ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [53ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [55ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [57ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [57ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [131ms] refreshing cache from master, [131ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [134ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@1503499965(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@7157078c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@284718322 after retry RPC@1239548409) -> (continuation of Deferred@1165424210 after retry RPC@482532044) -> (continuation of Deferred@769932068 after retry RPC@670732526), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@68e624b0 -> wakeup thread Time-limited test -> (continuation of Deferred@284718322 after retry RPC@1239548409) -> (continuation of Deferred@1165424210 after retry RPC@482532044) -> (continuation of Deferred@769932068 after retry RPC@670732526))) after lookup
14:00:34.326 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] connecting to peer
14:00:34.327 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:34.327 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] cleaning up while in state CONNECTING due to: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:34.327 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:34.405 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:34.405 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)[L],948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871),ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)]}]
14:00:34.406 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=10, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=216), Traces: [0ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [1ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [1ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [14ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [16ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [18ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [18ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [34ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [36ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [38ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [38ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [50ms] refreshing cache from master, [51ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [53ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [55ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [57ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [57ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [131ms] refreshing cache from master, [131ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [134ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [135ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [137ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [137ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [211ms] refreshing cache from master, [211ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [214ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@1503499965(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@7157078c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@284718322 after retry RPC@1239548409) -> (continuation of Deferred@1165424210 after retry RPC@482532044) -> (continuation of Deferred@769932068 after retry RPC@670732526) -> (continuation of Deferred@495011733 after retry RPC@930100122), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@68e624b0 -> wakeup thread Time-limited test -> (continuation of Deferred@284718322 after retry RPC@1239548409) -> (continuation of Deferred@1165424210 after retry RPC@482532044) -> (continuation of Deferred@769932068 after retry RPC@670732526) -> (continuation of Deferred@495011733 after retry RPC@930100122))) after lookup
14:00:34.407 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] connecting to peer
14:00:34.408 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:34.408 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] cleaning up while in state CONNECTING due to: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:34.408 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:34.625 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:34.625 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)[L],948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871),ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)]}]
14:00:34.625 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=12, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=435), Traces: [0ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [1ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [1ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [14ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [16ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [18ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [18ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [34ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [36ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [38ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [38ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [50ms] refreshing cache from master, [51ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [53ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [55ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [57ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [57ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [131ms] refreshing cache from master, [131ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [134ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [135ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [137ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [137ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [211ms] refreshing cache from master, [211ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [214ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [217ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [218ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [218ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [431ms] refreshing cache from master, [431ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [434ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@1503499965(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@7157078c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@284718322 after retry RPC@1239548409) -> (continuation of Deferred@1165424210 after retry RPC@482532044) -> (continuation of Deferred@769932068 after retry RPC@670732526) -> (continuation of Deferred@495011733 after retry RPC@930100122) -> (continuation of Deferred@667173184 after retry RPC@295590593), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@68e624b0 -> wakeup thread Time-limited test -> (continuation of Deferred@284718322 after retry RPC@1239548409) -> (continuation of Deferred@1165424210 after retry RPC@482532044) -> (continuation of Deferred@769932068 after retry RPC@670732526) -> (continuation of Deferred@495011733 after retry RPC@930100122) -> (continuation of Deferred@667173184 after retry RPC@295590593))) after lookup
14:00:34.626 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] connecting to peer
14:00:34.627 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:34.627 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] cleaning up while in state CONNECTING due to: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:34.627 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:35.005 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:35.005 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)[L],948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871),ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)]}]
14:00:35.006 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=14, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=816), Traces: [0ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [1ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [1ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [14ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [16ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [18ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [18ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [34ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [36ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [38ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [38ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [50ms] refreshing cache from master, [51ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [53ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [55ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [57ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [57ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [131ms] refreshing cache from master, [131ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [134ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [135ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [137ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [137ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [211ms] refreshing cache from master, [211ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [214ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [217ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [218ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [218ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [431ms] refreshing cache from master, [431ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [434ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [436ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [437ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [437ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [811ms] refreshing cache from master, [811ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [814ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@1503499965(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@7157078c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@284718322 after retry RPC@1239548409) -> (continuation of Deferred@1165424210 after retry RPC@482532044) -> (continuation of Deferred@769932068 after retry RPC@670732526) -> (continuation of Deferred@495011733 after retry RPC@930100122) -> (continuation of Deferred@667173184 after retry RPC@295590593) -> (continuation of Deferred@1959565915 after retry RPC@392662609), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@68e624b0 -> wakeup thread Time-limited test -> (continuation of Deferred@284718322 after retry RPC@1239548409) -> (continuation of Deferred@1165424210 after retry RPC@482532044) -> (continuation of Deferred@769932068 after retry RPC@670732526) -> (continuation of Deferred@495011733 after retry RPC@930100122) -> (continuation of Deferred@667173184 after retry RPC@295590593) -> (continuation of Deferred@1959565915 after retry RPC@392662609))) after lookup
14:00:35.006 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] connecting to peer
14:00:35.008 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:35.008 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] cleaning up while in state CONNECTING due to: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:35.008 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221
14:00:35.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:35.340783 18451 raft_consensus.cc:493] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 52f4ec85820b4a5897f1e992099d8c08)
14:00:35.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:35.340984 18451 raft_consensus.cc:515] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:35.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:35.341821 18451 leader_election.cc:290] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871), 52f4ec85820b4a5897f1e992099d8c08 (127.16.10.193:46221)
14:00:35.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:35.344794 18128 leader_election.cc:336] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 52f4ec85820b4a5897f1e992099d8c08 (127.16.10.193:46221): Network error: Client connection negotiation failed: client connection to 127.16.10.193:46221: connect: Connection refused (error 111)
14:00:35.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:35.351223 18392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf8e0efa2228413d8e2421c76198d284" candidate_uuid: "ad837748684c4c93a572185eda10fa14" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "948ae7bedcae4f2ea8a1825119425668" is_pre_election: true
14:00:35.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:35.351655 18392 raft_consensus.cc:2468] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad837748684c4c93a572185eda10fa14 in term 7.
14:00:35.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:35.352761 18128 leader_election.cc:304] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 948ae7bedcae4f2ea8a1825119425668, ad837748684c4c93a572185eda10fa14; no voters: 52f4ec85820b4a5897f1e992099d8c08
14:00:35.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:35.353071 18451 raft_consensus.cc:2804] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 8 FOLLOWER]: Leader pre-election won for term 9
14:00:35.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:35.353191 18451 raft_consensus.cc:493] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 52f4ec85820b4a5897f1e992099d8c08)
14:00:35.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:35.353288 18451 raft_consensus.cc:3060] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 8 FOLLOWER]: Advancing to term 9
14:00:35.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:35.355245 18451 raft_consensus.cc:515] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:35.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:35.355857 18451 leader_election.cc:290] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [CANDIDATE]: Term 9 election: Requested vote from peers 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871), 52f4ec85820b4a5897f1e992099d8c08 (127.16.10.193:46221)
14:00:35.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:35.356355 18392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf8e0efa2228413d8e2421c76198d284" candidate_uuid: "ad837748684c4c93a572185eda10fa14" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "948ae7bedcae4f2ea8a1825119425668"
14:00:35.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:35.356609 18392 raft_consensus.cc:3060] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 7 FOLLOWER]: Advancing to term 9
14:00:35.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:35.357311 18128 leader_election.cc:336] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 52f4ec85820b4a5897f1e992099d8c08 (127.16.10.193:46221): Network error: Client connection negotiation failed: client connection to 127.16.10.193:46221: connect: Connection refused (error 111)
14:00:35.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:35.360709 18392 raft_consensus.cc:2468] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad837748684c4c93a572185eda10fa14 in term 9.
14:00:35.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:35.361274 18128 leader_election.cc:304] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 948ae7bedcae4f2ea8a1825119425668, ad837748684c4c93a572185eda10fa14; no voters: 52f4ec85820b4a5897f1e992099d8c08
14:00:35.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:35.361620 18451 raft_consensus.cc:2804] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 9 FOLLOWER]: Leader election won for term 9
14:00:35.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:35.362143 18451 raft_consensus.cc:697] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 9 LEADER]: Becoming Leader. State: Replica: ad837748684c4c93a572185eda10fa14, State: Running, Role: LEADER
14:00:35.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:35.362747 18451 consensus_queue.cc:237] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 8.32, Last appended by leader: 32, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:35.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:35.366820 16469 catalog_manager.cc:5696] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 reported cstate change: term changed from 8 to 9, leader changed from 52f4ec85820b4a5897f1e992099d8c08 (127.16.10.193) to ad837748684c4c93a572185eda10fa14 (127.16.10.194). New cstate: current_term: 9 leader_uuid: "ad837748684c4c93a572185eda10fa14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } health_report { overall_health: UNKNOWN } } }
14:00:35.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:35.778455 18128 consensus_peers.cc:597] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 -> Peer 52f4ec85820b4a5897f1e992099d8c08 (127.16.10.193:46221): Couldn't send request to peer 52f4ec85820b4a5897f1e992099d8c08. Status: Network error: Client connection negotiation failed: client connection to 127.16.10.193:46221: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:00:35.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:35.985095 18392 raft_consensus.cc:1275] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 9 FOLLOWER]: Refusing update from remote peer ad837748684c4c93a572185eda10fa14: Log matching property violated. Preceding OpId in replica: term: 7 index: 28. Preceding OpId from leader: term: 9 index: 33. (index mismatch)
14:00:35.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:35.986119 18451 consensus_queue.cc:1048] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 28, Time since last communication: 0.000s
14:00:36.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:36.711606 18128 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.10.193:46221: connect: Connection refused (error 111) [suppressed 4 similar messages]
14:00:38.104 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:38.105 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221),948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871),ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)[L]]}]
14:00:38.105 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=16, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=3915), Traces: [0ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [1ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [1ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [14ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [16ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [18ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [18ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [34ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [36ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [38ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [38ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [50ms] refreshing cache from master, [51ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [53ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [55ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [57ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [57ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [131ms] refreshing cache from master, [131ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [134ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [135ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [137ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [137ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [211ms] refreshing cache from master, [211ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [214ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [217ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [218ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [218ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [431ms] refreshing cache from master, [431ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [434ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [436ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [437ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [437ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [811ms] refreshing cache from master, [811ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [814ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [816ms] sending RPC to server 52f4ec85820b4a5897f1e992099d8c08, [818ms] received response from server 52f4ec85820b4a5897f1e992099d8c08: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [818ms] delaying RPC due to: Network error: Failed to connect to peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221): Connection refused: /127.16.10.193:46221, [3911ms] refreshing cache from master, [3911ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [3914ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@1503499965(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@7157078c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@284718322 after retry RPC@1239548409) -> (continuation of Deferred@1165424210 after retry RPC@482532044) -> (continuation of Deferred@769932068 after retry RPC@670732526) -> (continuation of Deferred@495011733 after retry RPC@930100122) -> (continuation of Deferred@667173184 after retry RPC@295590593) -> (continuation of Deferred@1959565915 after retry RPC@392662609) -> (continuation of Deferred@2029262720 after retry RPC@1397361609), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@68e624b0 -> wakeup thread Time-limited test -> (continuation of Deferred@284718322 after retry RPC@1239548409) -> (continuation of Deferred@1165424210 after retry RPC@482532044) -> (continuation of Deferred@769932068 after retry RPC@670732526) -> (continuation of Deferred@495011733 after retry RPC@930100122) -> (continuation of Deferred@667173184 after retry RPC@295590593) -> (continuation of Deferred@1959565915 after retry RPC@392662609) -> (continuation of Deferred@2029262720 after retry RPC@1397361609))) after lookup
14:00:38.106 [DEBUG - kudu-nio-2] (Connection.java:748) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] connecting to peer
14:00:38.107 [DEBUG - kudu-nio-6] (Connection.java:757) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] Successfully connected to peer
14:00:38.122 [DEBUG - kudu-nio-6] (Negotiator.java:919) Authenticated connection [id: 0x79b6b68d, L:/127.0.0.1:36230 - R:/127.16.10.194:44603] using TOKEN/null
14:00:38.150 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.16.10.193:46221
14:00:38.150 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

14:00:38.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.151021 16427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu
14:00:38.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu
14:00:38.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/wal
14:00:38.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/data
14:00:38.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:00:38.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:00:38.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:00:38.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:00:38.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:00:38.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:00:38.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/logs
14:00:38.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/data/info.pb
14:00:38.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:00:38.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:00:38.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:00:38.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:00:38.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:00:38.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:00:38.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:00:38.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:00:38.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:00:38.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:00:38.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.10.193:46221
14:00:38.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.10.193
14:00:38.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.10.254:37885,127.16.10.253:33517,127.16.10.252:36395
14:00:38.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44113
14:00:38.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.10.193
14:00:38.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.10.212:33969
14:00:38.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:00:38.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:00:38.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:00:38.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:38.321674 18128 consensus_peers.cc:597] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 -> Peer 52f4ec85820b4a5897f1e992099d8c08 (127.16.10.193:46221): Couldn't send request to peer 52f4ec85820b4a5897f1e992099d8c08. Status: Network error: Client connection negotiation failed: client connection to 127.16.10.193:46221: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
14:00:38.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:38.504055 18489 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:00:38.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:38.504375 18489 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:00:38.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:38.504454 18489 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:00:38.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:38.513589 18489 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:00:38.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:38.513787 18489 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.10.193
14:00:38.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.524698 18489 tablet_server_runner.cc:78] Tablet server non-default flags:
14:00:38.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:00:38.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.10.212:33969
14:00:38.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:00:38.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:00:38.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/data
14:00:38.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/wal
14:00:38.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:00:38.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:00:38.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.10.193:46221
14:00:38.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:00:38.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:00:38.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:00:38.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/data/info.pb
14:00:38.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.10.193
14:00:38.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=44113
14:00:38.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.10.254:37885,127.16.10.253:33517,127.16.10.252:36395
14:00:38.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:00:38.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:00:38.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:00:38.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:00:38.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:00:38.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.10.193
14:00:38.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/logs
14:00:38.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:00:38.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:00:38.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:00:38.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:00:38.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:00:38.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
14:00:38.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:00:38.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 13:43:23 UTC on bdcb31816ec0
14:00:38.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11958
14:00:38.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
14:00:38.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.526433 18489 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:00:38.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.528398 18489 file_cache.cc:492] Constructed file cache file cache with capacity 419430
14:00:38.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:38.539072 18497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:00:38.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:38.539364 18499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:00:38.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:38.539330 18496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:00:38.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.540407 18489 server_base.cc:1061] running on GCE node
14:00:38.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.541059 18489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:00:38.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.542135 18489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:00:38.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.543385 18489 hybrid_clock.cc:648] HybridClock initialized: now 1779112838543289 us; error 79 us; skew 500 ppm
14:00:38.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.543713 18489 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:00:38.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.546070 18489 webserver.cc:533] Webserver started at http://127.16.10.193:44113/ using document root <none> and password file <none>
14:00:38.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.546944 18489 fs_manager.cc:362] Metadata directory not provided
14:00:38.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.547068 18489 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:00:38.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.553233 18489 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.000s
14:00:38.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.556315 18505 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:00:38.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.558054 18489 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
14:00:38.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.558275 18489 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/data,/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/wal
14:00:38.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "52f4ec85820b4a5897f1e992099d8c08"
14:00:38.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:00:07 on dist-test-slave-rq0s"
14:00:38.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.559161 18489 fs_report.cc:389] FS layout report
14:00:38.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:00:38.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/wal
14:00:38.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/wal
14:00:38.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/data/data
14:00:38.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:00:38.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:00:38.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:00:38.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:00:38.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:00:38.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:00:38.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:00:38.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:00:38.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:00:38.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:00:38.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:00:38.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.573055 18489 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:00:38.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.573976 18489 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:00:38.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.574231 18489 kserver.cc:163] Server-wide thread pool size limit: 3276
14:00:38.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.575290 18489 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:00:38.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.577486 18512 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
14:00:38.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.583730 18489 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:00:38.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.583844 18489 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.007s	user 0.001s	sys 0.000s
14:00:38.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.583990 18489 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:00:38.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.587102 18489 ts_tablet_manager.cc:616] Registered 1 tablets
14:00:38.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.587210 18489 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.003s	sys 0.000s
14:00:38.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.587756 18512 tablet_bootstrap.cc:492] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08: Bootstrap starting.
14:00:38.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.619310 18512 log.cc:826] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08: Log is configured to *not* fsync() on all Append() calls
14:00:38.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.638718 18489 rpc_server.cc:307] RPC server started. Bound to: 127.16.10.193:46221
14:00:38.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.638773 18619 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.10.193:46221 every 8 connection(s)
14:00:38.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.640237 18489 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-0/data/info.pb
14:00:38.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.650230 16427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu as pid 18489
14:00:38.650 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:00:38.681 [DEBUG - kudu-nio-6] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 27 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244, tablet=cf8e0efa2228413d8e2421c76198d284, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:00:38.681 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.16.10.194:44603
14:00:38.682 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

14:00:38.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.682271 16427 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu pid 18112
14:00:38.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.696156 18512 tablet_bootstrap.cc:492] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=24 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:00:38.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.696980 18512 tablet_bootstrap.cc:492] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08: Bootstrap complete.
14:00:38.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.701139 18620 heartbeater.cc:344] Connected to a master server at 127.16.10.252:36395
14:00:38.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.701570 18620 heartbeater.cc:461] Registering TS with master...
14:00:38.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.700380 18512 ts_tablet_manager.cc:1403] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08: Time spent bootstrapping tablet: real 0.113s	user 0.053s	sys 0.026s
14:00:38.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.702942 18620 heartbeater.cc:507] Master 127.16.10.252:36395 requested a full tablet report, sending...
14:00:38.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.700707 18621 heartbeater.cc:344] Connected to a master server at 127.16.10.254:37885
14:00:38.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.703334 18621 heartbeater.cc:461] Registering TS with master...
14:00:38.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.703961 18621 heartbeater.cc:507] Master 127.16.10.254:37885 requested a full tablet report, sending...
14:00:38.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.709836 18512 raft_consensus.cc:359] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:38.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.710810 18512 raft_consensus.cc:740] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f4ec85820b4a5897f1e992099d8c08, State: Initialized, Role: FOLLOWER
14:00:38.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.711544 18512 consensus_queue.cc:260] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 8.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:38.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.713253 18512 ts_tablet_manager.cc:1434] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08: Time spent starting tablet: real 0.011s	user 0.009s	sys 0.000s
14:00:38.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.714983 16468 ts_manager.cc:194] Re-registered known tserver with Master: 52f4ec85820b4a5897f1e992099d8c08 (127.16.10.193:46221)
14:00:38.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.715112 18622 heartbeater.cc:344] Connected to a master server at 127.16.10.253:33517
14:00:38.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.715263 18622 heartbeater.cc:461] Registering TS with master...
14:00:38.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.715811 18622 heartbeater.cc:507] Master 127.16.10.253:33517 requested a full tablet report, sending...
14:00:38.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.716566 16468 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.16.10.193:43347
14:00:38.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.717199 16611 ts_manager.cc:194] Re-registered known tserver with Master: 52f4ec85820b4a5897f1e992099d8c08 (127.16.10.193:46221)
14:00:38.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.717206 16539 ts_manager.cc:194] Re-registered known tserver with Master: 52f4ec85820b4a5897f1e992099d8c08 (127.16.10.193:46221)
14:00:38.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.720777 18236 generic_service.cc:196] Checking for leaks (request via RPC)
14:00:38.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.722711 18621 heartbeater.cc:499] Master 127.16.10.254:37885 was elected leader, sending a full tablet report...
14:00:38.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:38.830914 16427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu with pid 18112
14:00:38.849 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:00:38.849 [DEBUG - kudu-nio-6] (Connection.java:239) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] handling channelInactive
14:00:38.850 [DEBUG - kudu-nio-6] (Connection.java:709) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] cleaning up while in state READY due to: connection closed
14:00:38.850 [DEBUG - Time-limited test] (Connection.java:748) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] connecting to peer
14:00:38.851 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:38.851 [DEBUG - kudu-nio-7] (Connection.java:709) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] cleaning up while in state CONNECTING due to: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:38.851 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location ad837748684c4c93a572185eda10fa14(127.16.10.194:44603) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:38.864 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:38.865 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221),948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871),ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)[L]]}]
14:00:38.865 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=2, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=15), Traces: [0ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [1ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [1ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [14ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@1848780678(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@6da57dca -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5774a47e -> wakeup thread Time-limited test)) after lookup
14:00:38.865 [DEBUG - kudu-nio-2] (Connection.java:748) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] connecting to peer
14:00:38.866 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:38.866 [DEBUG - kudu-nio-0] (Connection.java:709) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] cleaning up while in state CONNECTING due to: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:38.867 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location ad837748684c4c93a572185eda10fa14(127.16.10.194:44603) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:38.884 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:38.884 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221),948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871),ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)[L]]}]
14:00:38.884 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=4, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=34), Traces: [0ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [1ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [1ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [14ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [15ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [16ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [17ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [34ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@1848780678(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@6da57dca -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1266244763 after retry RPC@970737374), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5774a47e -> wakeup thread Time-limited test -> (continuation of Deferred@1266244763 after retry RPC@970737374))) after lookup
14:00:38.885 [DEBUG - kudu-nio-2] (Connection.java:748) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] connecting to peer
14:00:38.886 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:38.886 [DEBUG - kudu-nio-1] (Connection.java:709) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] cleaning up while in state CONNECTING due to: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:38.886 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location ad837748684c4c93a572185eda10fa14(127.16.10.194:44603) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:38.923 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:38.924 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221),948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871),ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)[L]]}]
14:00:38.924 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=6, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=74), Traces: [0ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [1ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [1ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [14ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [15ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [16ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [17ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [34ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [35ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [36ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [37ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [70ms] refreshing cache from master, [71ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [73ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@1848780678(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@6da57dca -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1266244763 after retry RPC@970737374) -> (continuation of Deferred@1341656372 after retry RPC@348739747), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5774a47e -> wakeup thread Time-limited test -> (continuation of Deferred@1266244763 after retry RPC@970737374) -> (continuation of Deferred@1341656372 after retry RPC@348739747))) after lookup
14:00:38.924 [DEBUG - kudu-nio-2] (Connection.java:748) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] connecting to peer
14:00:38.925 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:38.925 [DEBUG - kudu-nio-2] (Connection.java:709) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] cleaning up while in state CONNECTING due to: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:38.925 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location ad837748684c4c93a572185eda10fa14(127.16.10.194:44603) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:39.024 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:39.024 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221),948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871),ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)[L]]}]
14:00:39.024 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=8, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=174), Traces: [0ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [1ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [1ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [14ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [15ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [16ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [17ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [34ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [35ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [36ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [37ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [70ms] refreshing cache from master, [71ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [73ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [74ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [75ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [75ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [171ms] refreshing cache from master, [171ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [174ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@1848780678(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@6da57dca -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1266244763 after retry RPC@970737374) -> (continuation of Deferred@1341656372 after retry RPC@348739747) -> (continuation of Deferred@1715536964 after retry RPC@440693245), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5774a47e -> wakeup thread Time-limited test -> (continuation of Deferred@1266244763 after retry RPC@970737374) -> (continuation of Deferred@1341656372 after retry RPC@348739747) -> (continuation of Deferred@1715536964 after retry RPC@440693245))) after lookup
14:00:39.025 [DEBUG - kudu-nio-2] (Connection.java:748) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] connecting to peer
14:00:39.026 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:39.026 [DEBUG - kudu-nio-3] (Connection.java:709) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] cleaning up while in state CONNECTING due to: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:39.027 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location ad837748684c4c93a572185eda10fa14(127.16.10.194:44603) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:39.043 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:39.044 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221),948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871),ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)[L]]}]
14:00:39.044 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=10, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=194), Traces: [0ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [1ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [1ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [14ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [15ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [16ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [17ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [34ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [35ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [36ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [37ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [70ms] refreshing cache from master, [71ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [73ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [74ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [75ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [75ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [171ms] refreshing cache from master, [171ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [174ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [175ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [176ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [177ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [190ms] refreshing cache from master, [190ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [193ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@1848780678(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@6da57dca -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1266244763 after retry RPC@970737374) -> (continuation of Deferred@1341656372 after retry RPC@348739747) -> (continuation of Deferred@1715536964 after retry RPC@440693245) -> (continuation of Deferred@951548520 after retry RPC@21315354), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5774a47e -> wakeup thread Time-limited test -> (continuation of Deferred@1266244763 after retry RPC@970737374) -> (continuation of Deferred@1341656372 after retry RPC@348739747) -> (continuation of Deferred@1715536964 after retry RPC@440693245) -> (continuation of Deferred@951548520 after retry RPC@21315354))) after lookup
14:00:39.044 [DEBUG - kudu-nio-2] (Connection.java:748) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] connecting to peer
14:00:39.045 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:39.045 [DEBUG - kudu-nio-4] (Connection.java:709) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] cleaning up while in state CONNECTING due to: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:39.046 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location ad837748684c4c93a572185eda10fa14(127.16.10.194:44603) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603
14:00:40.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:40.048843 18635 raft_consensus.cc:493] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ad837748684c4c93a572185eda10fa14)
14:00:40.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:40.049042 18635 raft_consensus.cc:515] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:40.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:40.050637 18635 leader_election.cc:290] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ad837748684c4c93a572185eda10fa14 (127.16.10.194:44603), 52f4ec85820b4a5897f1e992099d8c08 (127.16.10.193:46221)
14:00:40.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:40.052946 18325 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.10.194:44603: connect: Connection refused (error 111)
14:00:40.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:40.054602 18325 leader_election.cc:336] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ad837748684c4c93a572185eda10fa14 (127.16.10.194:44603): Network error: Client connection negotiation failed: client connection to 127.16.10.194:44603: connect: Connection refused (error 111)
14:00:40.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:40.059667 18574 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf8e0efa2228413d8e2421c76198d284" candidate_uuid: "948ae7bedcae4f2ea8a1825119425668" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "52f4ec85820b4a5897f1e992099d8c08" is_pre_election: true
14:00:40.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:40.060081 18574 raft_consensus.cc:2468] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948ae7bedcae4f2ea8a1825119425668 in term 8.
14:00:40.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:40.061034 18324 leader_election.cc:304] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52f4ec85820b4a5897f1e992099d8c08, 948ae7bedcae4f2ea8a1825119425668; no voters: ad837748684c4c93a572185eda10fa14
14:00:40.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:40.061477 18635 raft_consensus.cc:2804] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 9 FOLLOWER]: Leader pre-election won for term 10
14:00:40.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:40.061615 18635 raft_consensus.cc:493] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 9 FOLLOWER]: Starting leader election (detected failure of leader ad837748684c4c93a572185eda10fa14)
14:00:40.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:40.061719 18635 raft_consensus.cc:3060] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 9 FOLLOWER]: Advancing to term 10
14:00:40.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:40.063809 18635 raft_consensus.cc:515] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:40.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:40.064512 18635 leader_election.cc:290] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [CANDIDATE]: Term 10 election: Requested vote from peers ad837748684c4c93a572185eda10fa14 (127.16.10.194:44603), 52f4ec85820b4a5897f1e992099d8c08 (127.16.10.193:46221)
14:00:40.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:40.065093 18574 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf8e0efa2228413d8e2421c76198d284" candidate_uuid: "948ae7bedcae4f2ea8a1825119425668" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "52f4ec85820b4a5897f1e992099d8c08"
14:00:40.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:40.065301 18574 raft_consensus.cc:3060] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 8 FOLLOWER]: Advancing to term 10
14:00:40.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:40.066033 18325 leader_election.cc:336] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer ad837748684c4c93a572185eda10fa14 (127.16.10.194:44603): Network error: Client connection negotiation failed: client connection to 127.16.10.194:44603: connect: Connection refused (error 111)
14:00:40.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:40.069160 18574 raft_consensus.cc:2468] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948ae7bedcae4f2ea8a1825119425668 in term 10.
14:00:40.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:40.069675 18324 leader_election.cc:304] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52f4ec85820b4a5897f1e992099d8c08, 948ae7bedcae4f2ea8a1825119425668; no voters: ad837748684c4c93a572185eda10fa14
14:00:40.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:40.070024 18635 raft_consensus.cc:2804] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 10 FOLLOWER]: Leader election won for term 10
14:00:40.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:40.070598 18635 raft_consensus.cc:697] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 10 LEADER]: Becoming Leader. State: Replica: 948ae7bedcae4f2ea8a1825119425668, State: Running, Role: LEADER
14:00:40.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:40.071199 18635 consensus_queue.cc:237] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 9.36, Last appended by leader: 36, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:40.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:40.075155 16468 catalog_manager.cc:5696] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 reported cstate change: term changed from 9 to 10, leader changed from ad837748684c4c93a572185eda10fa14 (127.16.10.194) to 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195). New cstate: current_term: 10 leader_uuid: "948ae7bedcae4f2ea8a1825119425668" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } health_report { overall_health: UNKNOWN } } }
14:00:40.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:40.516753 18325 consensus_peers.cc:597] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 -> Peer ad837748684c4c93a572185eda10fa14 (127.16.10.194:44603): Couldn't send request to peer ad837748684c4c93a572185eda10fa14. Status: Network error: Client connection negotiation failed: client connection to 127.16.10.194:44603: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:00:40.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:40.530429 18574 raft_consensus.cc:1275] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 10 FOLLOWER]: Refusing update from remote peer 948ae7bedcae4f2ea8a1825119425668: Log matching property violated. Preceding OpId in replica: term: 8 index: 32. Preceding OpId from leader: term: 10 index: 37. (index mismatch)
14:00:40.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:40.531572 18635 consensus_queue.cc:1048] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 32, Time since last communication: 0.000s
14:00:41.104 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:41.104 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221),948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)[L],ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)]}]
14:00:41.105 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=12, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=2255), Traces: [0ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [1ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [1ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [14ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [15ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [16ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [17ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [34ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [35ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [36ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [37ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [70ms] refreshing cache from master, [71ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [73ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [74ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [75ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [75ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [171ms] refreshing cache from master, [171ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [174ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [175ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [176ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [177ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [190ms] refreshing cache from master, [190ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [193ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [194ms] sending RPC to server ad837748684c4c93a572185eda10fa14, [195ms] received response from server ad837748684c4c93a572185eda10fa14: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [196ms] delaying RPC due to: Network error: Failed to connect to peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603): Connection refused: /127.16.10.194:44603, [2251ms] refreshing cache from master, [2251ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [2254ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@1848780678(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@6da57dca -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1266244763 after retry RPC@970737374) -> (continuation of Deferred@1341656372 after retry RPC@348739747) -> (continuation of Deferred@1715536964 after retry RPC@440693245) -> (continuation of Deferred@951548520 after retry RPC@21315354) -> (continuation of Deferred@1546425209 after retry RPC@1178240599), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5774a47e -> wakeup thread Time-limited test -> (continuation of Deferred@1266244763 after retry RPC@970737374) -> (continuation of Deferred@1341656372 after retry RPC@348739747) -> (continuation of Deferred@1715536964 after retry RPC@440693245) -> (continuation of Deferred@951548520 after retry RPC@21315354) -> (continuation of Deferred@1546425209 after retry RPC@1178240599))) after lookup
14:00:41.106 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] connecting to peer
14:00:41.107 [DEBUG - kudu-nio-5] (Connection.java:757) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] Successfully connected to peer
14:00:41.126 [DEBUG - kudu-nio-5] (Negotiator.java:919) Authenticated connection [id: 0x45363d58, L:/127.0.0.1:42738 - R:/127.16.10.195:40871] using TOKEN/null
14:00:41.155 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.16.10.194:44603
14:00:41.155 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

14:00:41.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.156303 16427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu
14:00:41.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu
14:00:41.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/wal
14:00:41.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/data
14:00:41.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:00:41.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:00:41.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:00:41.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:00:41.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:00:41.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:00:41.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/logs
14:00:41.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/data/info.pb
14:00:41.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:00:41.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:00:41.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:00:41.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:00:41.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:00:41.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:00:41.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:00:41.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:00:41.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:00:41.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:00:41.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.10.194:44603
14:00:41.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.10.194
14:00:41.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.10.254:37885,127.16.10.253:33517,127.16.10.252:36395
14:00:41.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=33241
14:00:41.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.10.194
14:00:41.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.10.212:33969
14:00:41.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:00:41.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:00:41.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:00:41.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:41.522529 18666 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:00:41.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:41.522903 18666 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:00:41.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:41.522990 18666 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:00:41.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:41.532584 18666 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:00:41.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:41.532802 18666 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.10.194
14:00:41.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.543092 18666 tablet_server_runner.cc:78] Tablet server non-default flags:
14:00:41.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:00:41.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.10.212:33969
14:00:41.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:00:41.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:00:41.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/data
14:00:41.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/wal
14:00:41.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:00:41.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:00:41.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.10.194:44603
14:00:41.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:00:41.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:00:41.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:00:41.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/data/info.pb
14:00:41.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.10.194
14:00:41.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=33241
14:00:41.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.10.254:37885,127.16.10.253:33517,127.16.10.252:36395
14:00:41.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:00:41.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:00:41.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:00:41.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:00:41.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:00:41.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.10.194
14:00:41.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/logs
14:00:41.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:00:41.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:00:41.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:00:41.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:00:41.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:00:41.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
14:00:41.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:00:41.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 13:43:23 UTC on bdcb31816ec0
14:00:41.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11958
14:00:41.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
14:00:41.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.544924 18666 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:00:41.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.546901 18666 file_cache.cc:492] Constructed file cache file cache with capacity 419430
14:00:41.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:41.559275 18671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:00:41.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:41.560150 18672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:00:41.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:41.560999 18674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:00:41.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.562132 18666 server_base.cc:1061] running on GCE node
14:00:41.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.562937 18666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:00:41.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.564157 18666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:00:41.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.565438 18666 hybrid_clock.cc:648] HybridClock initialized: now 1779112841565369 us; error 84 us; skew 500 ppm
14:00:41.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.565908 18666 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:00:41.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.568898 18666 webserver.cc:533] Webserver started at http://127.16.10.194:33241/ using document root <none> and password file <none>
14:00:41.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.569756 18666 fs_manager.cc:362] Metadata directory not provided
14:00:41.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.569850 18666 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:00:41.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.575920 18666 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.004s	sys 0.000s
14:00:41.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.579161 18680 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:00:41.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.580953 18666 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.002s
14:00:41.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.581182 18666 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/data,/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/wal
14:00:41.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "ad837748684c4c93a572185eda10fa14"
14:00:41.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:00:08 on dist-test-slave-rq0s"
14:00:41.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.581949 18666 fs_report.cc:389] FS layout report
14:00:41.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:00:41.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/wal
14:00:41.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/wal
14:00:41.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/data/data
14:00:41.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:00:41.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:00:41.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:00:41.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:00:41.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:00:41.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:00:41.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:00:41.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:00:41.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:00:41.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:00:41.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:00:41.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.595651 18666 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:00:41.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.596544 18666 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:00:41.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.596794 18666 kserver.cc:163] Server-wide thread pool size limit: 3276
14:00:41.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.597762 18666 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:00:41.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.600059 18687 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
14:00:41.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.606209 18666 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:00:41.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.606309 18666 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.007s	user 0.001s	sys 0.000s
14:00:41.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.606416 18666 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:00:41.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.609676 18666 ts_tablet_manager.cc:616] Registered 1 tablets
14:00:41.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.609833 18666 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.001s	sys 0.000s
14:00:41.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.610371 18687 tablet_bootstrap.cc:492] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14: Bootstrap starting.
14:00:41.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.639331 18687 log.cc:826] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14: Log is configured to *not* fsync() on all Append() calls
14:00:41.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.675436 18666 rpc_server.cc:307] RPC server started. Bound to: 127.16.10.194:44603
14:00:41.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.675552 18794 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.10.194:44603 every 8 connection(s)
14:00:41.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.677088 18666 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-1/data/info.pb
14:00:41.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.687394 16427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu as pid 18666
14:00:41.688 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:00:41.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.692328 18687 tablet_bootstrap.cc:492] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14: Bootstrap replayed 1/1 log segments. Stats: ops{read=36 overwritten=0 applied=36 ignored=0} inserts{seen=27 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:00:41.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.693184 18687 tablet_bootstrap.cc:492] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14: Bootstrap complete.
14:00:41.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.695083 18687 ts_tablet_manager.cc:1403] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14: Time spent bootstrapping tablet: real 0.085s	user 0.047s	sys 0.031s
14:00:41.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.702175 18687 raft_consensus.cc:359] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 9 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:41.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.703372 18687 raft_consensus.cc:740] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad837748684c4c93a572185eda10fa14, State: Initialized, Role: FOLLOWER
14:00:41.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.704427 18687 consensus_queue.cc:260] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 9.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:41.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.707842 18796 heartbeater.cc:344] Connected to a master server at 127.16.10.254:37885
14:00:41.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.708006 18795 heartbeater.cc:344] Connected to a master server at 127.16.10.252:36395
14:00:41.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.708321 18796 heartbeater.cc:461] Registering TS with master...
14:00:41.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.708329 18795 heartbeater.cc:461] Registering TS with master...
14:00:41.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.709712 18796 heartbeater.cc:507] Master 127.16.10.254:37885 requested a full tablet report, sending...
14:00:41.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.710003 18795 heartbeater.cc:507] Master 127.16.10.252:36395 requested a full tablet report, sending...
14:00:41.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.713071 16468 ts_manager.cc:194] Re-registered known tserver with Master: ad837748684c4c93a572185eda10fa14 (127.16.10.194:44603)
14:00:41.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.713212 16611 ts_manager.cc:194] Re-registered known tserver with Master: ad837748684c4c93a572185eda10fa14 (127.16.10.194:44603)
14:00:41.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.714681 16468 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.16.10.194:42073
14:00:41.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.717242 18687 ts_tablet_manager.cc:1434] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14: Time spent starting tablet: real 0.022s	user 0.011s	sys 0.001s
14:00:41.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.718607 18797 heartbeater.cc:344] Connected to a master server at 127.16.10.253:33517
14:00:41.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.718760 18797 heartbeater.cc:461] Registering TS with master...
14:00:41.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.719162 18797 heartbeater.cc:507] Master 127.16.10.253:33517 requested a full tablet report, sending...
14:00:41.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.720258 18796 heartbeater.cc:499] Master 127.16.10.254:37885 was elected leader, sending a full tablet report...
14:00:41.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.721117 16540 ts_manager.cc:194] Re-registered known tserver with Master: ad837748684c4c93a572185eda10fa14 (127.16.10.194:44603)
14:00:41.724 [DEBUG - kudu-nio-5] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 30 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244, tablet=cf8e0efa2228413d8e2421c76198d284, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:00:41.730 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.16.10.195:40871
14:00:41.731 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

14:00:41.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.731261 16427 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu pid 18308
14:00:41.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.741109 18432 generic_service.cc:196] Checking for leaks (request via RPC)
14:00:41.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:41.859401 16427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu with pid 18308
14:00:41.879 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:00:41.879 [DEBUG - kudu-nio-5] (Connection.java:239) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] handling channelInactive
14:00:41.879 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] cleaning up while in state READY due to: connection closed
14:00:41.880 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871) for tablet cf8e0efa2228413d8e2421c76198d284: connection closed
14:00:41.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:41.880533 18810 negotiation.cc:336] Failed RPC negotiation. Trace:
14:00:41.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:00:41.859168 (+     0us) reactor.cc:678] Submitting negotiation task for server connection from 127.16.10.195:60339 (local address 127.16.10.194:44603)
14:00:41.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:00:41.859788 (+   620us) server_negotiation.cc:207] Beginning negotiation
14:00:41.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:00:41.859799 (+    11us) server_negotiation.cc:400] Waiting for connection header
14:00:41.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:00:41.859906 (+   107us) server_negotiation.cc:408] Connection header received
14:00:41.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:00:41.860036 (+   130us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
14:00:41.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:00:41.860042 (+     6us) server_negotiation.cc:462] Received NEGOTIATE request from client
14:00:41.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:00:41.860262 (+   220us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
14:00:41.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:00:41.860438 (+   176us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
14:00:41.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:00:41.878803 (+ 18365us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.10.195:60339: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
14:00:41.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":336,"threads_started":1}
14:00:41.905 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:41.905 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221),948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)[L],ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)]}]
14:00:41.905 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=2, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=26), Traces: [0ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [1ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [25ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@1206820828(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@55f4b148 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@41ebfa81 -> wakeup thread Time-limited test)) after lookup
14:00:41.906 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] connecting to peer
14:00:41.907 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:41.907 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] cleaning up while in state CONNECTING due to: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:41.907 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:41.924 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:41.924 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221),948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)[L],ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)]}]
14:00:41.924 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=4, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=45), Traces: [0ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [1ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [25ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [26ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [28ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [28ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [42ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [45ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@1206820828(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@55f4b148 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1991502140 after retry RPC@1746025113), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@41ebfa81 -> wakeup thread Time-limited test -> (continuation of Deferred@1991502140 after retry RPC@1746025113))) after lookup
14:00:41.925 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] connecting to peer
14:00:41.926 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:41.926 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] cleaning up while in state CONNECTING due to: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:41.927 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:41.964 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:41.965 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221),948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)[L],ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)]}]
14:00:41.965 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=6, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=85), Traces: [0ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [1ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [25ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [26ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [28ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [28ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [42ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [45ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [46ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [47ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [48ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [82ms] refreshing cache from master, [82ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [85ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@1206820828(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@55f4b148 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1991502140 after retry RPC@1746025113) -> (continuation of Deferred@700599147 after retry RPC@1358785080), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@41ebfa81 -> wakeup thread Time-limited test -> (continuation of Deferred@1991502140 after retry RPC@1746025113) -> (continuation of Deferred@700599147 after retry RPC@1358785080))) after lookup
14:00:41.965 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] connecting to peer
14:00:41.966 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:41.966 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] cleaning up while in state CONNECTING due to: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:41.967 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:42.104 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:42.104 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221),948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)[L],ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)]}]
14:00:42.104 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=8, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=225), Traces: [0ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [1ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [25ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [26ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [28ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [28ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [42ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [45ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [46ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [47ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [48ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [82ms] refreshing cache from master, [82ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [85ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [86ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [88ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [88ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [221ms] refreshing cache from master, [222ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [224ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@1206820828(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@55f4b148 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1991502140 after retry RPC@1746025113) -> (continuation of Deferred@700599147 after retry RPC@1358785080) -> (continuation of Deferred@287772034 after retry RPC@707795252), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@41ebfa81 -> wakeup thread Time-limited test -> (continuation of Deferred@1991502140 after retry RPC@1746025113) -> (continuation of Deferred@700599147 after retry RPC@1358785080) -> (continuation of Deferred@287772034 after retry RPC@707795252))) after lookup
14:00:42.105 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] connecting to peer
14:00:42.106 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:42.107 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] cleaning up while in state CONNECTING due to: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:42.107 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:42.224 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:42.224 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221),948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)[L],ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)]}]
14:00:42.225 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=10, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=345), Traces: [0ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [1ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [25ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [26ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [28ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [28ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [42ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [45ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [46ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [47ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [48ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [82ms] refreshing cache from master, [82ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [85ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [86ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [88ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [88ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [221ms] refreshing cache from master, [222ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [224ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [226ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [228ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [228ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [342ms] refreshing cache from master, [342ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [345ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@1206820828(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@55f4b148 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1991502140 after retry RPC@1746025113) -> (continuation of Deferred@700599147 after retry RPC@1358785080) -> (continuation of Deferred@287772034 after retry RPC@707795252) -> (continuation of Deferred@1015959167 after retry RPC@383584611), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@41ebfa81 -> wakeup thread Time-limited test -> (continuation of Deferred@1991502140 after retry RPC@1746025113) -> (continuation of Deferred@700599147 after retry RPC@1358785080) -> (continuation of Deferred@287772034 after retry RPC@707795252) -> (continuation of Deferred@1015959167 after retry RPC@383584611))) after lookup
14:00:42.225 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] connecting to peer
14:00:42.226 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:42.226 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] cleaning up while in state CONNECTING due to: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:42.226 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871) for tablet cf8e0efa2228413d8e2421c76198d284: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871
14:00:42.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:42.923777 18811 raft_consensus.cc:493] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
14:00:42.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:42.924052 18811 raft_consensus.cc:515] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:42.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:42.925706 18811 leader_election.cc:290] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871), 52f4ec85820b4a5897f1e992099d8c08 (127.16.10.193:46221)
14:00:42.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:42.928308 18682 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.10.195:40871: connect: Connection refused (error 111)
14:00:42.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:42.929867 18682 leader_election.cc:336] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871): Network error: Client connection negotiation failed: client connection to 127.16.10.195:40871: connect: Connection refused (error 111)
14:00:42.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:42.933266 18574 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf8e0efa2228413d8e2421c76198d284" candidate_uuid: "ad837748684c4c93a572185eda10fa14" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "52f4ec85820b4a5897f1e992099d8c08" is_pre_election: true
14:00:42.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:42.934140 18682 leader_election.cc:304] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad837748684c4c93a572185eda10fa14; no voters: 52f4ec85820b4a5897f1e992099d8c08, 948ae7bedcae4f2ea8a1825119425668
14:00:42.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:42.934549 18811 raft_consensus.cc:2749] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
14:00:43.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:43.154033 18816 raft_consensus.cc:493] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 948ae7bedcae4f2ea8a1825119425668)
14:00:43.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:43.154332 18816 raft_consensus.cc:515] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:43.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:43.156080 18816 leader_election.cc:290] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ad837748684c4c93a572185eda10fa14 (127.16.10.194:44603), 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871)
14:00:43.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:43.157729 18507 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.10.195:40871: connect: Connection refused (error 111)
14:00:43.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:43.159953 18507 leader_election.cc:336] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871): Network error: Client connection negotiation failed: client connection to 127.16.10.195:40871: connect: Connection refused (error 111)
14:00:43.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:43.165849 18750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf8e0efa2228413d8e2421c76198d284" candidate_uuid: "52f4ec85820b4a5897f1e992099d8c08" candidate_term: 11 candidate_status { last_received { term: 10 index: 40 } } ignore_live_leader: false dest_uuid: "ad837748684c4c93a572185eda10fa14" is_pre_election: true
14:00:43.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:43.166399 18750 raft_consensus.cc:2468] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f4ec85820b4a5897f1e992099d8c08 in term 9.
14:00:43.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:43.167667 18508 leader_election.cc:304] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52f4ec85820b4a5897f1e992099d8c08, ad837748684c4c93a572185eda10fa14; no voters: 948ae7bedcae4f2ea8a1825119425668
14:00:43.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:43.168219 18816 raft_consensus.cc:2804] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 10 FOLLOWER]: Leader pre-election won for term 11
14:00:43.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:43.168376 18816 raft_consensus.cc:493] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 948ae7bedcae4f2ea8a1825119425668)
14:00:43.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:43.168519 18816 raft_consensus.cc:3060] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 10 FOLLOWER]: Advancing to term 11
14:00:43.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:43.170925 18816 raft_consensus.cc:515] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:43.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:43.171700 18816 leader_election.cc:290] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [CANDIDATE]: Term 11 election: Requested vote from peers ad837748684c4c93a572185eda10fa14 (127.16.10.194:44603), 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871)
14:00:43.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:43.172421 18750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf8e0efa2228413d8e2421c76198d284" candidate_uuid: "52f4ec85820b4a5897f1e992099d8c08" candidate_term: 11 candidate_status { last_received { term: 10 index: 40 } } ignore_live_leader: false dest_uuid: "ad837748684c4c93a572185eda10fa14"
14:00:43.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:43.172678 18750 raft_consensus.cc:3060] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 9 FOLLOWER]: Advancing to term 11
14:00:43.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:43.174331 18507 leader_election.cc:336] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871): Network error: Client connection negotiation failed: client connection to 127.16.10.195:40871: connect: Connection refused (error 111)
14:00:43.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:43.176729 18750 raft_consensus.cc:2468] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f4ec85820b4a5897f1e992099d8c08 in term 11.
14:00:43.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:43.177258 18508 leader_election.cc:304] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52f4ec85820b4a5897f1e992099d8c08, ad837748684c4c93a572185eda10fa14; no voters: 948ae7bedcae4f2ea8a1825119425668
14:00:43.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:43.177587 18816 raft_consensus.cc:2804] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 11 FOLLOWER]: Leader election won for term 11
14:00:43.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:43.178133 18816 raft_consensus.cc:697] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [term 11 LEADER]: Becoming Leader. State: Replica: 52f4ec85820b4a5897f1e992099d8c08, State: Running, Role: LEADER
14:00:43.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:43.178733 18816 consensus_queue.cc:237] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 10.40, Last appended by leader: 40, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:43.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:43.183020 16468 catalog_manager.cc:5696] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 reported cstate change: term changed from 10 to 11, leader changed from 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195) to 52f4ec85820b4a5897f1e992099d8c08 (127.16.10.193). New cstate: current_term: 11 leader_uuid: "52f4ec85820b4a5897f1e992099d8c08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } health_report { overall_health: HEALTHY } } }
14:00:43.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:43.611171 18750 raft_consensus.cc:1275] T cf8e0efa2228413d8e2421c76198d284 P ad837748684c4c93a572185eda10fa14 [term 11 FOLLOWER]: Refusing update from remote peer 52f4ec85820b4a5897f1e992099d8c08: Log matching property violated. Preceding OpId in replica: term: 9 index: 36. Preceding OpId from leader: term: 11 index: 41. (index mismatch)
14:00:43.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:43.612236 18816 consensus_queue.cc:1048] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 36, Time since last communication: 0.000s
14:00:43.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:43.626587 18507 consensus_peers.cc:597] T cf8e0efa2228413d8e2421c76198d284 P 52f4ec85820b4a5897f1e992099d8c08 -> Peer 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871): Couldn't send request to peer 948ae7bedcae4f2ea8a1825119425668. Status: Network error: Client connection negotiation failed: client connection to 127.16.10.195:40871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:00:44.024 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet cf8e0efa2228413d8e2421c76198d284 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244' with partition [<start>, <end>)
14:00:44.024 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=cf8e0efa2228413d8e2421c76198d284@[52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)[L],948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871),ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)]}]
14:00:44.024 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=cf8e0efa2228413d8e2421c76198d284, attempt=12, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=2144), Traces: [0ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [1ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [22ms] refreshing cache from master, [22ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [25ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [26ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [28ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [28ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [42ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [45ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [46ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [47ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [48ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [82ms] refreshing cache from master, [82ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [85ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [86ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [88ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [88ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [221ms] refreshing cache from master, [222ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [224ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [226ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [228ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [228ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [342ms] refreshing cache from master, [342ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [345ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, [346ms] sending RPC to server 948ae7bedcae4f2ea8a1825119425668, [347ms] received response from server 948ae7bedcae4f2ea8a1825119425668: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [347ms] delaying RPC due to: Network error: Failed to connect to peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871): Connection refused: /127.16.10.195:40871, [2141ms] refreshing cache from master, [2142ms] Sub RPC GetTableLocations: sending RPC to server master-127.16.10.254:37885, [2144ms] Sub RPC GetTableLocations: received response from server master-127.16.10.254:37885: OK, deferred=Deferred@1206820828(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$243/0x00007f5c943371b0@55f4b148 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1991502140 after retry RPC@1746025113) -> (continuation of Deferred@700599147 after retry RPC@1358785080) -> (continuation of Deferred@287772034 after retry RPC@707795252) -> (continuation of Deferred@1015959167 after retry RPC@383584611) -> (continuation of Deferred@1076468365 after retry RPC@1039320906), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@41ebfa81 -> wakeup thread Time-limited test -> (continuation of Deferred@1991502140 after retry RPC@1746025113) -> (continuation of Deferred@700599147 after retry RPC@1358785080) -> (continuation of Deferred@287772034 after retry RPC@707795252) -> (continuation of Deferred@1015959167 after retry RPC@383584611) -> (continuation of Deferred@1076468365 after retry RPC@1039320906))) after lookup
14:00:44.025 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] connecting to peer
14:00:44.026 [DEBUG - kudu-nio-3] (Connection.java:757) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] Successfully connected to peer
14:00:44.046 [DEBUG - kudu-nio-3] (Negotiator.java:919) Authenticated connection [id: 0x93326ccf, L:/127.0.0.1:45494 - R:/127.16.10.193:46221] using TOKEN/null
14:00:44.070 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.16.10.195:40871
14:00:44.071 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

14:00:44.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.071915 16427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu
14:00:44.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu
14:00:44.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/wal
14:00:44.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/data
14:00:44.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:00:44.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:00:44.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:00:44.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:00:44.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:00:44.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:00:44.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/logs
14:00:44.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/data/info.pb
14:00:44.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:00:44.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:00:44.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:00:44.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:00:44.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:00:44.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:00:44.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:00:44.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:00:44.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:00:44.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:00:44.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.10.195:40871
14:00:44.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.10.195
14:00:44.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.10.254:37885,127.16.10.253:33517,127.16.10.252:36395
14:00:44.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45985
14:00:44.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.10.195
14:00:44.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.10.212:33969
14:00:44.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:00:44.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:00:44.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:00:44.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:44.419875 18844 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:00:44.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:44.420358 18844 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:00:44.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:44.420444 18844 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:00:44.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:44.429100 18844 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:00:44.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:44.429282 18844 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.10.195
14:00:44.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.439924 18844 tablet_server_runner.cc:78] Tablet server non-default flags:
14:00:44.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:00:44.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.16.10.212:33969
14:00:44.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:00:44.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:00:44.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/data
14:00:44.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/wal
14:00:44.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:00:44.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:00:44.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.16.10.195:40871
14:00:44.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:00:44.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:00:44.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:00:44.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/data/info.pb
14:00:44.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.16.10.195
14:00:44.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45985
14:00:44.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.16.10.254:37885,127.16.10.253:33517,127.16.10.252:36395
14:00:44.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:00:44.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:00:44.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:00:44.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:00:44.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:00:44.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.16.10.195
14:00:44.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/logs
14:00:44.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:00:44.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:00:44.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:00:44.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:00:44.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:00:44.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
14:00:44.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:00:44.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 13:43:23 UTC on bdcb31816ec0
14:00:44.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11958
14:00:44.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) ASAN enabled
14:00:44.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.441613 18844 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:00:44.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.443568 18844 file_cache.cc:492] Constructed file cache file cache with capacity 419430
14:00:44.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:44.454119 18852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:00:44.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:44.454242 18851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:00:44.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:44.454788 18854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:00:44.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.455332 18844 server_base.cc:1061] running on GCE node
14:00:44.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.455905 18844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:00:44.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.456879 18844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:00:44.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.458146 18844 hybrid_clock.cc:648] HybridClock initialized: now 1779112844458065 us; error 69 us; skew 500 ppm
14:00:44.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.458575 18844 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:00:44.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.460842 18844 webserver.cc:533] Webserver started at http://127.16.10.195:45985/ using document root <none> and password file <none>
14:00:44.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.461597 18844 fs_manager.cc:362] Metadata directory not provided
14:00:44.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.461688 18844 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:00:44.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.467846 18844 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.004s	sys 0.000s
14:00:44.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.471347 18860 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:00:44.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.472975 18844 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.000s
14:00:44.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.473135 18844 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/data,/tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/wal
14:00:44.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "948ae7bedcae4f2ea8a1825119425668"
14:00:44.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:00:08 on dist-test-slave-rq0s"
14:00:44.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.473846 18844 fs_report.cc:389] FS layout report
14:00:44.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:00:44.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/wal
14:00:44.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/wal
14:00:44.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/data/data
14:00:44.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:00:44.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:00:44.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:00:44.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:00:44.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:00:44.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:00:44.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:00:44.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:00:44.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:00:44.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:00:44.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:00:44.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.511721 18844 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:00:44.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.512669 18844 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:00:44.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.512934 18844 kserver.cc:163] Server-wide thread pool size limit: 3276
14:00:44.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.513895 18844 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:00:44.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.516112 18867 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
14:00:44.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.522804 18844 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:00:44.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.522905 18844 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.007s	user 0.001s	sys 0.000s
14:00:44.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.523010 18844 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:00:44.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.525959 18844 ts_tablet_manager.cc:616] Registered 1 tablets
14:00:44.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.526070 18844 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s	user 0.000s	sys 0.000s
14:00:44.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.526752 18867 tablet_bootstrap.cc:492] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668: Bootstrap starting.
14:00:44.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.561595 18867 log.cc:826] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668: Log is configured to *not* fsync() on all Append() calls
14:00:44.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.576588 18844 rpc_server.cc:307] RPC server started. Bound to: 127.16.10.195:40871
14:00:44.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.576704 18974 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.10.195:40871 every 8 connection(s)
14:00:44.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.578538 18844 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskw_3vRI/test-tmp/mini-kudu-cluster17710449030038628694/ts-2/data/info.pb
14:00:44.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.581110 16427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu as pid 18844
14:00:44.581 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:00:44.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.642943 18867 tablet_bootstrap.cc:492] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668: Bootstrap replayed 1/1 log segments. Stats: ops{read=40 overwritten=0 applied=40 ignored=0} inserts{seen=30 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:00:44.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.643966 18867 tablet_bootstrap.cc:492] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668: Bootstrap complete.
14:00:44.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.646277 18867 ts_tablet_manager.cc:1403] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668: Time spent bootstrapping tablet: real 0.120s	user 0.088s	sys 0.015s
14:00:44.652 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244, tablet=cf8e0efa2228413d8e2421c76198d284, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:00:44.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.655725 18867 raft_consensus.cc:359] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 10 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:44.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.657080 18867 raft_consensus.cc:740] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 10 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948ae7bedcae4f2ea8a1825119425668, State: Initialized, Role: FOLLOWER
14:00:44.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.658159 18867 consensus_queue.cc:260] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 10.40, Last appended by leader: 40, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad837748684c4c93a572185eda10fa14" member_type: VOTER last_known_addr { host: "127.16.10.194" port: 44603 } } peers { permanent_uuid: "948ae7bedcae4f2ea8a1825119425668" member_type: VOTER last_known_addr { host: "127.16.10.195" port: 40871 } } peers { permanent_uuid: "52f4ec85820b4a5897f1e992099d8c08" member_type: VOTER last_known_addr { host: "127.16.10.193" port: 46221 } }
14:00:44.662 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779112809244, tablet=cf8e0efa2228413d8e2421c76198d284, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:00:44.668 [DEBUG - main] (Connection.java:575) [peer master-127.16.10.252:36395(127.16.10.252:36395)] disconnecting while in state READY
14:00:44.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.670243 18867 ts_tablet_manager.cc:1434] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668: Time spent starting tablet: real 0.024s	user 0.019s	sys 0.004s
14:00:44.671 [DEBUG - main] (Connection.java:575) [peer master-127.16.10.253:33517(127.16.10.253:33517)] disconnecting while in state READY
14:00:44.672 [DEBUG - main] (Connection.java:575) [peer 948ae7bedcae4f2ea8a1825119425668(127.16.10.195:40871)] disconnecting while in state TERMINATED
14:00:44.672 [DEBUG - main] (Connection.java:575) [peer master-127.16.10.254:37885(127.16.10.254:37885)] disconnecting while in state READY
14:00:44.672 [DEBUG - main] (Connection.java:575) [peer ad837748684c4c93a572185eda10fa14(127.16.10.194:44603)] disconnecting while in state TERMINATED
14:00:44.672 [DEBUG - main] (Connection.java:575) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] disconnecting while in state READY
14:00:44.673 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.16.10.252:36395(127.16.10.252:36395)] handling channelInactive
14:00:44.674 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.16.10.252:36395(127.16.10.252:36395)] cleaning up while in state READY due to: connection closed
14:00:44.674 [DEBUG - kudu-nio-3] (Connection.java:239) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] handling channelInactive
14:00:44.674 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 52f4ec85820b4a5897f1e992099d8c08(127.16.10.193:46221)] cleaning up while in state READY due to: connection closed
14:00:44.674 [DEBUG - kudu-nio-2] (Connection.java:239) [peer master-127.16.10.254:37885(127.16.10.254:37885)] handling channelInactive
14:00:44.674 [DEBUG - kudu-nio-2] (Connection.java:709) [peer master-127.16.10.254:37885(127.16.10.254:37885)] cleaning up while in state READY due to: connection closed
14:00:44.683 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2704) Releasing all remaining resources
14:00:44.684 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.16.10.253:33517(127.16.10.253:33517)] handling channelInactive
14:00:44.684 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.16.10.253:33517(127.16.10.253:33517)] cleaning up while in state READY due to: connection closed
14:00:44.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.684819 16427 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu pid 18489
14:00:44.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.691231 18975 heartbeater.cc:344] Connected to a master server at 127.16.10.252:36395
14:00:44.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.691618 18975 heartbeater.cc:461] Registering TS with master...
14:00:44.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.692948 18975 heartbeater.cc:507] Master 127.16.10.252:36395 requested a full tablet report, sending...
14:00:44.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.700201 16610 ts_manager.cc:194] Re-registered known tserver with Master: 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871)
14:00:44.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.711792 18929 raft_consensus.cc:3060] T cf8e0efa2228413d8e2421c76198d284 P 948ae7bedcae4f2ea8a1825119425668 [term 10 FOLLOWER]: Advancing to term 11
14:00:44.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.722357 18976 heartbeater.cc:344] Connected to a master server at 127.16.10.254:37885
14:00:44.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.722589 18976 heartbeater.cc:461] Registering TS with master...
14:00:44.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.723244 18976 heartbeater.cc:507] Master 127.16.10.254:37885 requested a full tablet report, sending...
14:00:44.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.732165 18977 heartbeater.cc:344] Connected to a master server at 127.16.10.253:33517
14:00:44.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.732401 18977 heartbeater.cc:461] Registering TS with master...
14:00:44.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.733135 18977 heartbeater.cc:507] Master 127.16.10.253:33517 requested a full tablet report, sending...
14:00:44.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.746512 18615 generic_service.cc:196] Checking for leaks (request via RPC)
14:00:44.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.755643 16469 ts_manager.cc:194] Re-registered known tserver with Master: 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871)
14:00:44.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.755859 16540 ts_manager.cc:194] Re-registered known tserver with Master: 948ae7bedcae4f2ea8a1825119425668 (127.16.10.195:40871)
14:00:44.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.757859 16469 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.16.10.195:49567
14:00:44.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.760391 18976 heartbeater.cc:499] Master 127.16.10.254:37885 was elected leader, sending a full tablet report...
14:00:44.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.865944 16427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu with pid 18489
14:00:44.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.884531 16427 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu pid 18666
14:00:44.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.892313 18790 generic_service.cc:196] Checking for leaks (request via RPC)
14:00:44.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:44.987685 16427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu with pid 18666
14:00:45.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:45.005656 16427 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu pid 18844
14:00:45.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:45.012004 18970 generic_service.cc:196] Checking for leaks (request via RPC)
14:00:45.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:45.108346 16427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu with pid 18844
14:00:45.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:45.126434 16427 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu pid 16438
14:00:45.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:45.128144 16499 generic_service.cc:196] Checking for leaks (request via RPC)
14:00:45.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:45.253046 16427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu with pid 16438
14:00:45.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:00:45.276024 16597 connection.cc:582] server connection from 127.0.0.1:52488 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
14:00:45.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:45.276168 16427 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu pid 16507
14:00:45.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:45.277416 16570 generic_service.cc:196] Checking for leaks (request via RPC)
14:00:45.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:45.399024 16427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu with pid 16507
14:00:45.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:45.419471 16427 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu pid 16577
14:00:45.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:45.420699 16641 generic_service.cc:196] Checking for leaks (request via RPC)
14:00:45.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:00:45.539033 16427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskw_3vRI/build/asan/bin/kudu with pid 16577
14:00:45.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-05-18T14:00:45Z chronyd exiting

Time: 41.735

OK (1 test)

