JUnit version 4.13.2
.14:02:52.003 [DEBUG - main] (RandomUtils.java:49) Using random seed: 1786456971995
14:02:52.061 [INFO - main] (RetryRule.java:138) Creating RetryStatement with result reporter and retry count of 0 (no retries)
14:02:52.121 [INFO - main] (KuduTestHarness.java:155) Creating a new MiniKuduCluster...
14:02:52.133 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskhBFXjG/test-tmp
14:02:52.139 [INFO - main] (KuduBinaryLocator.java:83) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/release/bin
14:02:52.142 [INFO - main] (MiniKuduCluster.java:217) Starting process: [/tmp/dist-test-taskhBFXjG/java/build/dist-test/../../../build/release/bin/kudu, test, mini_cluster, --serialization=pb]
14:02:52.333 [DEBUG - main] (MiniKuduCluster.java:185) Request: create_cluster {
  num_masters: 3
  num_tservers: 3
  enable_kerberos: false
  cluster_root: "/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675"
  hms_mode: NONE
  mini_kdc_options {
  }
  principal: "kudu"
  mini_oidc_options {
  }
}

14:02:52.347 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
14:02:52.348 [DEBUG - main] (MiniKuduCluster.java:185) Request: start_cluster {
}

14:02:52.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.349854 10597 mini_oidc.cc:150] Starting JWKS server
14:02:52.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.350682 10597 webserver.cc:533] Webserver started at http://127.0.0.1:44501/ using document root <none> and password file <none>
14:02:52.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.350780 10597 mini_oidc.cc:186] Starting OIDC Discovery server
14:02:52.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.350957 10597 webserver.cc:533] Webserver started at http://127.0.0.1:41409/ using document root <none> and password file <none>
14:02:52.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-08-11T14:02:52Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
14:02:52.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-08-11T14:02:52Z Disabled control of system clock
14:02:52.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.365216 10597 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskhBFXjG/build/release/bin/kudu
14:02:52.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskhBFXjG/build/release/bin/kudu
14:02:52.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-0/wal
14:02:52.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-0/data
14:02:52.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:02:52.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:02:52.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:02:52.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:02:52.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:02:52.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:02:52.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-0/logs
14:02:52.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-0/data/info.pb
14:02:52.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:02:52.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:02:52.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:02:52.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:02:52.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:02:52.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:02:52.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:02:52.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:02:52.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
14:02:52.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:02:52.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
14:02:52.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
14:02:52.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.89.126:36189
14:02:52.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.89.126
14:02:52.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:02:52.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.89.84:33487
14:02:52.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:02:52.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:02:52.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:02:52.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
14:02:52.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.10.89.126:36189,127.10.89.125:42399,127.10.89.124:42407
14:02:52.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
14:02:52.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:44501/jwks.json with env {}
14:02:52.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.448042 10610 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:02:52.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.448287 10610 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:02:52.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.448321 10610 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:02:52.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.450381 10610 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
14:02:52.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.450464 10610 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:02:52.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.450485 10610 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
14:02:52.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.450505 10610 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
14:02:52.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.450522 10610 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
14:02:52.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.450541 10610 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:44501/jwks.json
14:02:52.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.452800 10610 master_runner.cc:387] Master server non-default flags:
14:02:52.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-0/logs
14:02:52.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:02:52.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:02:52.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:02:52.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.89.84:33487
14:02:52.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:02:52.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:02:52.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-0/data
14:02:52.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-0/wal
14:02:52.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
14:02:52.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.10.89.126:36189,127.10.89.125:42399,127.10.89.124:42407
14:02:52.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:02:52.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:02:52.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
14:02:52.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.89.126:36189
14:02:52.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
14:02:52.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:02:52.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
14:02:52.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:44501/jwks.json
14:02:52.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:02:52.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:02:52.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-0/data/info.pb
14:02:52.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.89.126
14:02:52.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:02:52.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:02:52.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.10610
14:02:52.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:02:52.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:02:52.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:02:52.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:02:52.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:02:52.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
14:02:52.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:02:52.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 4bcd631c449d3c6649730ca6e07660c8168f0d44
14:02:52.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
14:02:52.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Aug 2026 13:43:13 UTC on e7f111948823
14:02:52.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 13289
14:02:52.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.453088 10610 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:02:52.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.453400 10610 file_cache.cc:504] Constructed file cache file cache with capacity 419430
14:02:52.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.457304 10619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:02:52.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.457329 10615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:02:52.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.457424 10610 server_base.cc:1061] running on GCE node
14:02:52.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.457329 10616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:02:52.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.457778 10610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:02:52.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.458011 10610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:02:52.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.459144 10610 hybrid_clock.cc:648] HybridClock initialized: now 1786456972459061 us; error 98 us; skew 500 ppm
14:02:52.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.460520 10610 webserver.cc:533] Webserver started at http://127.10.89.126:39697/ using document root <none> and password file <none>
14:02:52.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.460687 10610 fs_manager.cc:362] Metadata directory not provided
14:02:52.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.460717 10610 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:02:52.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.460804 10610 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
14:02:52.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.461792 10610 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-0/data/instance:
14:02:52.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "155b8f49c1ea408b8f15e66b715e4bdf"
14:02:52.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 14:02:52 on dist-test-slave-g350"
14:02:52.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.462193 10610 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-0/wal/instance:
14:02:52.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "155b8f49c1ea408b8f15e66b715e4bdf"
14:02:52.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 14:02:52 on dist-test-slave-g350"
14:02:52.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.463397 10610 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.002s	sys 0.000s
14:02:52.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.464159 10625 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:02:52.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.464314 10610 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.001s	sys 0.000s
14:02:52.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.464390 10610 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-0/data,/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-0/wal
14:02:52.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "155b8f49c1ea408b8f15e66b715e4bdf"
14:02:52.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 14:02:52 on dist-test-slave-g350"
14:02:52.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.464433 10610 fs_report.cc:389] FS layout report
14:02:52.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:02:52.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-0/wal
14:02:52.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-0/wal
14:02:52.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-0/data/data
14:02:52.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:02:52.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:02:52.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:02:52.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:02:52.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:02:52.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:02:52.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:02:52.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:02:52.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:02:52.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:02:52.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:02:52.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.483665 10610 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:02:52.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.483939 10610 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:02:52.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.484022 10610 kserver.cc:163] Server-wide thread pool size limit: 3276
14:02:52.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.488279 10610 rpc_server.cc:307] RPC server started. Bound to: 127.10.89.126:36189
14:02:52.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.488314 10690 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.89.126:36189 every 8 connection(s)
14:02:52.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.488619 10610 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-0/data/info.pb
14:02:52.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.489264 10691 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
14:02:52.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.490548 10597 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskhBFXjG/build/release/bin/kudu as pid 10610
14:02:52.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.490677 10597 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-0/wal/instance
14:02:52.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.491672 10597 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskhBFXjG/build/release/bin/kudu
14:02:52.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskhBFXjG/build/release/bin/kudu
14:02:52.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-1/wal
14:02:52.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-1/data
14:02:52.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:02:52.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:02:52.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:02:52.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:02:52.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:02:52.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:02:52.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-1/logs
14:02:52.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-1/data/info.pb
14:02:52.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:02:52.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:02:52.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:02:52.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:02:52.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:02:52.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:02:52.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:02:52.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:02:52.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
14:02:52.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:02:52.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
14:02:52.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
14:02:52.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.89.125:42399
14:02:52.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.89.125
14:02:52.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:02:52.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.89.84:33487
14:02:52.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:02:52.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:02:52.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:02:52.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
14:02:52.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.10.89.126:36189,127.10.89.125:42399,127.10.89.124:42407
14:02:52.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
14:02:52.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:44501/jwks.json with env {}
14:02:52.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.491513 10691 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.89.126" port: 36189 } has no permanent_uuid. Determining permanent_uuid...
14:02:52.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.503865 10691 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.89.125" port: 42399 } has no permanent_uuid. Determining permanent_uuid...
14:02:52.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.504369 10628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.89.125:42399: connect: Connection refused (error 111)
14:02:52.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.504832 10691 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.10.89.125:42399: Network error: Client connection negotiation failed: client connection to 127.10.89.125:42399: connect: Connection refused (error 111)
14:02:52.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.554056 10691 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.89.125" port: 42399 } attempt: 1
14:02:52.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.554935 10691 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.10.89.125:42399: Network error: Client connection negotiation failed: client connection to 127.10.89.125:42399: connect: Connection refused (error 111)
14:02:52.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.575258 10696 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:02:52.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.575477 10696 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:02:52.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.575511 10696 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:02:52.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.577440 10696 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
14:02:52.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.577517 10696 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:02:52.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.577538 10696 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
14:02:52.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.577556 10696 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
14:02:52.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.577574 10696 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
14:02:52.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.577592 10696 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:44501/jwks.json
14:02:52.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.579749 10696 master_runner.cc:387] Master server non-default flags:
14:02:52.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-1/logs
14:02:52.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:02:52.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:02:52.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:02:52.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.89.84:33487
14:02:52.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:02:52.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:02:52.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-1/data
14:02:52.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-1/wal
14:02:52.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
14:02:52.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.10.89.126:36189,127.10.89.125:42399,127.10.89.124:42407
14:02:52.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:02:52.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:02:52.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
14:02:52.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.89.125:42399
14:02:52.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
14:02:52.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:02:52.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
14:02:52.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:44501/jwks.json
14:02:52.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:02:52.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:02:52.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-1/data/info.pb
14:02:52.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.89.125
14:02:52.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:02:52.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:02:52.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.10696
14:02:52.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:02:52.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:02:52.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:02:52.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:02:52.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:02:52.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
14:02:52.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:02:52.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 4bcd631c449d3c6649730ca6e07660c8168f0d44
14:02:52.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
14:02:52.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Aug 2026 13:43:13 UTC on e7f111948823
14:02:52.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 13289
14:02:52.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.580024 10696 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:02:52.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.580315 10696 file_cache.cc:504] Constructed file cache file cache with capacity 419430
14:02:52.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.583153 10704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:02:52.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.583338 10705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:02:52.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.583405 10708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:02:52.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.583545 10696 server_base.cc:1061] running on GCE node
14:02:52.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.583815 10696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:02:52.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.584087 10696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:02:52.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.585266 10696 hybrid_clock.cc:648] HybridClock initialized: now 1786456972585244 us; error 37 us; skew 500 ppm
14:02:52.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.586694 10696 webserver.cc:533] Webserver started at http://127.10.89.125:39057/ using document root <none> and password file <none>
14:02:52.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.586913 10696 fs_manager.cc:362] Metadata directory not provided
14:02:52.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.586967 10696 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:02:52.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.587087 10696 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
14:02:52.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.588250 10696 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-1/data/instance:
14:02:52.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "43052a37ca8648629532aac191629eb2"
14:02:52.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 14:02:52 on dist-test-slave-g350"
14:02:52.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.588609 10696 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-1/wal/instance:
14:02:52.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "43052a37ca8648629532aac191629eb2"
14:02:52.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 14:02:52 on dist-test-slave-g350"
14:02:52.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.590245 10696 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.002s	sys 0.001s
14:02:52.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.591149 10714 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:02:52.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.591307 10696 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.001s	sys 0.000s
14:02:52.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.591364 10696 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-1/data,/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-1/wal
14:02:52.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "43052a37ca8648629532aac191629eb2"
14:02:52.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 14:02:52 on dist-test-slave-g350"
14:02:52.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.591424 10696 fs_report.cc:389] FS layout report
14:02:52.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:02:52.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-1/wal
14:02:52.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-1/wal
14:02:52.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-1/data/data
14:02:52.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:02:52.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:02:52.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:02:52.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:02:52.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:02:52.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:02:52.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:02:52.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:02:52.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:02:52.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:02:52.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:02:52.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.610791 10696 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:02:52.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.611142 10696 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:02:52.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.611279 10696 kserver.cc:163] Server-wide thread pool size limit: 3276
14:02:52.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.615710 10696 rpc_server.cc:307] RPC server started. Bound to: 127.10.89.125:42399
14:02:52.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.615754 10770 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.89.125:42399 every 8 connection(s)
14:02:52.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.616580 10696 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-1/data/info.pb
14:02:52.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.617313 10771 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
14:02:52.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.617697 10597 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskhBFXjG/build/release/bin/kudu as pid 10696
14:02:52.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.617798 10597 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-1/wal/instance
14:02:52.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.618357 10597 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskhBFXjG/build/release/bin/kudu
14:02:52.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskhBFXjG/build/release/bin/kudu
14:02:52.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-2/wal
14:02:52.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-2/data
14:02:52.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:02:52.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:02:52.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:02:52.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:02:52.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:02:52.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:02:52.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-2/logs
14:02:52.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-2/data/info.pb
14:02:52.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:02:52.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:02:52.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:02:52.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:02:52.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:02:52.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:02:52.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:02:52.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:02:52.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
14:02:52.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:02:52.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
14:02:52.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
14:02:52.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.89.124:42407
14:02:52.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.89.124
14:02:52.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:02:52.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.89.84:33487
14:02:52.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:02:52.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:02:52.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:02:52.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
14:02:52.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.10.89.126:36189,127.10.89.125:42399,127.10.89.124:42407
14:02:52.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
14:02:52.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:44501/jwks.json with env {}
14:02:52.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.619509 10771 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.89.126" port: 36189 } has no permanent_uuid. Determining permanent_uuid...
14:02:52.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.623075 10691 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.89.125" port: 42399 } attempt: 2
14:02:52.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.626083 10771 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.89.125" port: 42399 } has no permanent_uuid. Determining permanent_uuid...
14:02:52.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.629752 10691 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.89.124" port: 42407 } has no permanent_uuid. Determining permanent_uuid...
14:02:52.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.630335 10691 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.10.89.124:42407: Network error: Client connection negotiation failed: client connection to 127.10.89.124:42407: connect: Connection refused (error 111)
14:02:52.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.631669 10771 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.89.124" port: 42407 } has no permanent_uuid. Determining permanent_uuid...
14:02:52.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.632010 10719 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.89.124:42407: connect: Connection refused (error 111)
14:02:52.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.633945 10771 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.10.89.124:42407: Network error: Client connection negotiation failed: client connection to 127.10.89.124:42407: connect: Connection refused (error 111)
14:02:52.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.673504 10691 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.89.124" port: 42407 } attempt: 1
14:02:52.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.674669 10691 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.10.89.124:42407: Network error: Client connection negotiation failed: client connection to 127.10.89.124:42407: connect: Connection refused (error 111)
14:02:52.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.683223 10771 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.89.124" port: 42407 } attempt: 1
14:02:52.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.684016 10771 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.10.89.124:42407: Network error: Client connection negotiation failed: client connection to 127.10.89.124:42407: connect: Connection refused (error 111)
14:02:52.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.705935 10773 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:02:52.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.706141 10773 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:02:52.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.706174 10773 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:02:52.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.708003 10773 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
14:02:52.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.708076 10773 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:02:52.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.708099 10773 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
14:02:52.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.708117 10773 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
14:02:52.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.708134 10773 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
14:02:52.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.708153 10773 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:44501/jwks.json
14:02:52.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.710297 10773 master_runner.cc:387] Master server non-default flags:
14:02:52.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-2/logs
14:02:52.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:02:52.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:02:52.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:02:52.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.89.84:33487
14:02:52.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:02:52.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:02:52.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-2/data
14:02:52.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-2/wal
14:02:52.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
14:02:52.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.10.89.126:36189,127.10.89.125:42399,127.10.89.124:42407
14:02:52.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:02:52.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:02:52.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
14:02:52.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.89.124:42407
14:02:52.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
14:02:52.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:02:52.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
14:02:52.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:44501/jwks.json
14:02:52.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:02:52.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:02:52.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-2/data/info.pb
14:02:52.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.89.124
14:02:52.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:02:52.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:02:52.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.10773
14:02:52.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:02:52.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:02:52.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:02:52.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:02:52.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:02:52.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
14:02:52.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:02:52.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 4bcd631c449d3c6649730ca6e07660c8168f0d44
14:02:52.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
14:02:52.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Aug 2026 13:43:13 UTC on e7f111948823
14:02:52.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 13289
14:02:52.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.710567 10773 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:02:52.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.710870 10773 file_cache.cc:504] Constructed file cache file cache with capacity 419430
14:02:52.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.713742 10786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:02:52.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.713776 10782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:02:52.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.713995 10784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:02:52.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.715183 10773 server_base.cc:1061] running on GCE node
14:02:52.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.715533 10773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:02:52.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.715808 10773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:02:52.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.716995 10773 hybrid_clock.cc:648] HybridClock initialized: now 1786456972716967 us; error 39 us; skew 500 ppm
14:02:52.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.718441 10773 webserver.cc:533] Webserver started at http://127.10.89.124:38271/ using document root <none> and password file <none>
14:02:52.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.718640 10773 fs_manager.cc:362] Metadata directory not provided
14:02:52.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.718683 10773 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:02:52.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.718801 10773 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
14:02:52.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.719918 10773 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-2/data/instance:
14:02:52.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d6bb4436c3c94a9c89ef988fe1e68010"
14:02:52.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 14:02:52 on dist-test-slave-g350"
14:02:52.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.720316 10773 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-2/wal/instance:
14:02:52.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d6bb4436c3c94a9c89ef988fe1e68010"
14:02:52.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 14:02:52 on dist-test-slave-g350"
14:02:52.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.721843 10691 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.89.124" port: 42407 } attempt: 2
14:02:52.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.721999 10773 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.003s	sys 0.000s
14:02:52.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.722874 10793 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:02:52.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.723100 10691 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.10.89.124:42407: Network error: Client connection negotiation failed: client connection to 127.10.89.124:42407: connect: Connection refused (error 111)
14:02:52.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.723591 10773 fs_manager.cc:730] Time spent opening block manager: real 0.001s	user 0.001s	sys 0.000s
14:02:52.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.723666 10773 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-2/data,/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-2/wal
14:02:52.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d6bb4436c3c94a9c89ef988fe1e68010"
14:02:52.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 14:02:52 on dist-test-slave-g350"
14:02:52.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.723724 10773 fs_report.cc:389] FS layout report
14:02:52.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:02:52.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-2/wal
14:02:52.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-2/wal
14:02:52.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-2/data/data
14:02:52.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:02:52.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:02:52.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:02:52.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:02:52.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:02:52.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:02:52.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:02:52.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:02:52.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:02:52.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:02:52.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:02:52.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.752239 10771 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.89.124" port: 42407 } attempt: 2
14:02:52.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.752370 10773 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:02:52.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.752698 10773 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:02:52.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.752840 10773 kserver.cc:163] Server-wide thread pool size limit: 3276
14:02:52.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.753199 10771 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.10.89.124:42407: Network error: Client connection negotiation failed: client connection to 127.10.89.124:42407: connect: Connection refused (error 111)
14:02:52.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.757323 10773 rpc_server.cc:307] RPC server started. Bound to: 127.10.89.124:42407
14:02:52.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.757359 10857 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.89.124:42407 every 8 connection(s)
14:02:52.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.757728 10773 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-2/data/info.pb
14:02:52.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.758350 10858 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
14:02:52.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.760044 10858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.89.126" port: 36189 } has no permanent_uuid. Determining permanent_uuid...
14:02:52.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.760257 10597 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskhBFXjG/build/release/bin/kudu as pid 10773
14:02:52.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.760327 10597 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/master-2/wal/instance
14:02:52.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.765116 10858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.89.125" port: 42399 } has no permanent_uuid. Determining permanent_uuid...
14:02:52.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.768569 10858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.89.124" port: 42407 } has no permanent_uuid. Determining permanent_uuid...
14:02:52.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.773080 10858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6bb4436c3c94a9c89ef988fe1e68010: Bootstrap starting.
14:02:52.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.773784 10858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6bb4436c3c94a9c89ef988fe1e68010: Neither blocks nor log segments found. Creating new log.
14:02:52.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.773991 10858 log.cc:826] T 00000000000000000000000000000000 P d6bb4436c3c94a9c89ef988fe1e68010: Log is configured to *not* fsync() on all Append() calls
14:02:52.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.774760 10858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6bb4436c3c94a9c89ef988fe1e68010: No bootstrap required, opened a new log
14:02:52.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.776688 10858 raft_consensus.cc:359] T 00000000000000000000000000000000 P d6bb4436c3c94a9c89ef988fe1e68010 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155b8f49c1ea408b8f15e66b715e4bdf" member_type: VOTER last_known_addr { host: "127.10.89.126" port: 36189 } } peers { permanent_uuid: "43052a37ca8648629532aac191629eb2" member_type: VOTER last_known_addr { host: "127.10.89.125" port: 42399 } } peers { permanent_uuid: "d6bb4436c3c94a9c89ef988fe1e68010" member_type: VOTER last_known_addr { host: "127.10.89.124" port: 42407 } }
14:02:52.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.776870 10858 raft_consensus.cc:385] T 00000000000000000000000000000000 P d6bb4436c3c94a9c89ef988fe1e68010 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
14:02:52.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.776912 10858 raft_consensus.cc:740] T 00000000000000000000000000000000 P d6bb4436c3c94a9c89ef988fe1e68010 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6bb4436c3c94a9c89ef988fe1e68010, State: Initialized, Role: FOLLOWER
14:02:52.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.777001 10858 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6bb4436c3c94a9c89ef988fe1e68010 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155b8f49c1ea408b8f15e66b715e4bdf" member_type: VOTER last_known_addr { host: "127.10.89.126" port: 36189 } } peers { permanent_uuid: "43052a37ca8648629532aac191629eb2" member_type: VOTER last_known_addr { host: "127.10.89.125" port: 42399 } } peers { permanent_uuid: "d6bb4436c3c94a9c89ef988fe1e68010" member_type: VOTER last_known_addr { host: "127.10.89.124" port: 42407 } }
14:02:52.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.777282 10858 sys_catalog.cc:565] T 00000000000000000000000000000000 P d6bb4436c3c94a9c89ef988fe1e68010 [sys.catalog]: configured and running, proceeding with master startup.
14:02:52.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.777315 10866 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6bb4436c3c94a9c89ef988fe1e68010 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155b8f49c1ea408b8f15e66b715e4bdf" member_type: VOTER last_known_addr { host: "127.10.89.126" port: 36189 } } peers { permanent_uuid: "43052a37ca8648629532aac191629eb2" member_type: VOTER last_known_addr { host: "127.10.89.125" port: 42399 } } peers { permanent_uuid: "d6bb4436c3c94a9c89ef988fe1e68010" member_type: VOTER last_known_addr { host: "127.10.89.124" port: 42407 } } }
14:02:52.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.777400 10866 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6bb4436c3c94a9c89ef988fe1e68010 [sys.catalog]: This master's current role is: FOLLOWER
14:02:52.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.779217 10879 catalog_manager.cc:1594] T 00000000000000000000000000000000 P d6bb4436c3c94a9c89ef988fe1e68010: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
14:02:52.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.779403 10879 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
14:02:52.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.779937 10773 master_runner.cc:187] Error getting master registration for 127.10.89.126:36189: OK, instance_id { permanent_uuid: "155b8f49c1ea408b8f15e66b715e4bdf" instance_seqno: 1786456972487161 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
14:02:52.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.781639 10773 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
14:02:52.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.830265 10691 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.89.124" port: 42407 } attempt: 3
14:02:52.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.834697 10691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 155b8f49c1ea408b8f15e66b715e4bdf: Bootstrap starting.
14:02:52.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.835355 10691 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 155b8f49c1ea408b8f15e66b715e4bdf: Neither blocks nor log segments found. Creating new log.
14:02:52.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.835572 10691 log.cc:826] T 00000000000000000000000000000000 P 155b8f49c1ea408b8f15e66b715e4bdf: Log is configured to *not* fsync() on all Append() calls
14:02:52.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.836377 10691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 155b8f49c1ea408b8f15e66b715e4bdf: No bootstrap required, opened a new log
14:02:52.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.837756 10691 raft_consensus.cc:359] T 00000000000000000000000000000000 P 155b8f49c1ea408b8f15e66b715e4bdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155b8f49c1ea408b8f15e66b715e4bdf" member_type: VOTER last_known_addr { host: "127.10.89.126" port: 36189 } } peers { permanent_uuid: "43052a37ca8648629532aac191629eb2" member_type: VOTER last_known_addr { host: "127.10.89.125" port: 42399 } } peers { permanent_uuid: "d6bb4436c3c94a9c89ef988fe1e68010" member_type: VOTER last_known_addr { host: "127.10.89.124" port: 42407 } }
14:02:52.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.837872 10691 raft_consensus.cc:385] T 00000000000000000000000000000000 P 155b8f49c1ea408b8f15e66b715e4bdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
14:02:52.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.837894 10691 raft_consensus.cc:740] T 00000000000000000000000000000000 P 155b8f49c1ea408b8f15e66b715e4bdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155b8f49c1ea408b8f15e66b715e4bdf, State: Initialized, Role: FOLLOWER
14:02:52.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.837999 10691 consensus_queue.cc:260] T 00000000000000000000000000000000 P 155b8f49c1ea408b8f15e66b715e4bdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155b8f49c1ea408b8f15e66b715e4bdf" member_type: VOTER last_known_addr { host: "127.10.89.126" port: 36189 } } peers { permanent_uuid: "43052a37ca8648629532aac191629eb2" member_type: VOTER last_known_addr { host: "127.10.89.125" port: 42399 } } peers { permanent_uuid: "d6bb4436c3c94a9c89ef988fe1e68010" member_type: VOTER last_known_addr { host: "127.10.89.124" port: 42407 } }
14:02:52.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.838284 10882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 155b8f49c1ea408b8f15e66b715e4bdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155b8f49c1ea408b8f15e66b715e4bdf" member_type: VOTER last_known_addr { host: "127.10.89.126" port: 36189 } } peers { permanent_uuid: "43052a37ca8648629532aac191629eb2" member_type: VOTER last_known_addr { host: "127.10.89.125" port: 42399 } } peers { permanent_uuid: "d6bb4436c3c94a9c89ef988fe1e68010" member_type: VOTER last_known_addr { host: "127.10.89.124" port: 42407 } } }
14:02:52.839 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.838335 10691 sys_catalog.cc:565] T 00000000000000000000000000000000 P 155b8f49c1ea408b8f15e66b715e4bdf [sys.catalog]: configured and running, proceeding with master startup.
14:02:52.839 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.838366 10882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 155b8f49c1ea408b8f15e66b715e4bdf [sys.catalog]: This master's current role is: FOLLOWER
14:02:52.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.840657 10894 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 155b8f49c1ea408b8f15e66b715e4bdf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
14:02:52.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.840956 10894 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
14:02:52.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.841392 10610 master_runner.cc:187] Error getting master registration for 127.10.89.125:42399: OK, instance_id { permanent_uuid: "43052a37ca8648629532aac191629eb2" instance_seqno: 1786456972614559 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
14:02:52.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.843146 10610 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
14:02:52.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.844369 10771 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.89.124" port: 42407 } attempt: 3
14:02:52.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.849433 10771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43052a37ca8648629532aac191629eb2: Bootstrap starting.
14:02:52.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.850226 10771 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43052a37ca8648629532aac191629eb2: Neither blocks nor log segments found. Creating new log.
14:02:52.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.850499 10771 log.cc:826] T 00000000000000000000000000000000 P 43052a37ca8648629532aac191629eb2: Log is configured to *not* fsync() on all Append() calls
14:02:52.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.851567 10771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43052a37ca8648629532aac191629eb2: No bootstrap required, opened a new log
14:02:52.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.853324 10771 raft_consensus.cc:359] T 00000000000000000000000000000000 P 43052a37ca8648629532aac191629eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155b8f49c1ea408b8f15e66b715e4bdf" member_type: VOTER last_known_addr { host: "127.10.89.126" port: 36189 } } peers { permanent_uuid: "43052a37ca8648629532aac191629eb2" member_type: VOTER last_known_addr { host: "127.10.89.125" port: 42399 } } peers { permanent_uuid: "d6bb4436c3c94a9c89ef988fe1e68010" member_type: VOTER last_known_addr { host: "127.10.89.124" port: 42407 } }
14:02:52.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.853444 10771 raft_consensus.cc:385] T 00000000000000000000000000000000 P 43052a37ca8648629532aac191629eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
14:02:52.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.853475 10771 raft_consensus.cc:740] T 00000000000000000000000000000000 P 43052a37ca8648629532aac191629eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43052a37ca8648629532aac191629eb2, State: Initialized, Role: FOLLOWER
14:02:52.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.853570 10771 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43052a37ca8648629532aac191629eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155b8f49c1ea408b8f15e66b715e4bdf" member_type: VOTER last_known_addr { host: "127.10.89.126" port: 36189 } } peers { permanent_uuid: "43052a37ca8648629532aac191629eb2" member_type: VOTER last_known_addr { host: "127.10.89.125" port: 42399 } } peers { permanent_uuid: "d6bb4436c3c94a9c89ef988fe1e68010" member_type: VOTER last_known_addr { host: "127.10.89.124" port: 42407 } }
14:02:52.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.853883 10771 sys_catalog.cc:565] T 00000000000000000000000000000000 P 43052a37ca8648629532aac191629eb2 [sys.catalog]: configured and running, proceeding with master startup.
14:02:52.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.854615 10897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43052a37ca8648629532aac191629eb2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155b8f49c1ea408b8f15e66b715e4bdf" member_type: VOTER last_known_addr { host: "127.10.89.126" port: 36189 } } peers { permanent_uuid: "43052a37ca8648629532aac191629eb2" member_type: VOTER last_known_addr { host: "127.10.89.125" port: 42399 } } peers { permanent_uuid: "d6bb4436c3c94a9c89ef988fe1e68010" member_type: VOTER last_known_addr { host: "127.10.89.124" port: 42407 } } }
14:02:52.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.854983 10897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43052a37ca8648629532aac191629eb2 [sys.catalog]: This master's current role is: FOLLOWER
14:02:52.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.855964 10908 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 43052a37ca8648629532aac191629eb2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
14:02:52.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.856021 10908 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
14:02:52.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.858542 10696 master_runner.cc:264] No leader master found from master 43052a37ca8648629532aac191629eb2
14:02:52.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.858588 10696 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
14:02:52.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.861956 10721 tablet.cc:2406] T 00000000000000000000000000000000 P 43052a37ca8648629532aac191629eb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
14:02:52.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.954080 10897 raft_consensus.cc:493] T 00000000000000000000000000000000 P 43052a37ca8648629532aac191629eb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
14:02:52.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.954855 10897 raft_consensus.cc:515] T 00000000000000000000000000000000 P 43052a37ca8648629532aac191629eb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155b8f49c1ea408b8f15e66b715e4bdf" member_type: VOTER last_known_addr { host: "127.10.89.126" port: 36189 } } peers { permanent_uuid: "43052a37ca8648629532aac191629eb2" member_type: VOTER last_known_addr { host: "127.10.89.125" port: 42399 } } peers { permanent_uuid: "d6bb4436c3c94a9c89ef988fe1e68010" member_type: VOTER last_known_addr { host: "127.10.89.124" port: 42407 } }
14:02:52.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.955734 10658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "43052a37ca8648629532aac191629eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155b8f49c1ea408b8f15e66b715e4bdf" is_pre_election: true
14:02:52.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.955897 10658 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 155b8f49c1ea408b8f15e66b715e4bdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43052a37ca8648629532aac191629eb2 in term 0.
14:02:52.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.956137 10719 leader_election.cc:304] T 00000000000000000000000000000000 P 43052a37ca8648629532aac191629eb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155b8f49c1ea408b8f15e66b715e4bdf, 43052a37ca8648629532aac191629eb2; no voters: 
14:02:52.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.957228 10828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "43052a37ca8648629532aac191629eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6bb4436c3c94a9c89ef988fe1e68010" is_pre_election: true
14:02:52.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.957389 10828 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d6bb4436c3c94a9c89ef988fe1e68010 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43052a37ca8648629532aac191629eb2 in term 0.
14:02:52.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.958082 10897 leader_election.cc:290] T 00000000000000000000000000000000 P 43052a37ca8648629532aac191629eb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 155b8f49c1ea408b8f15e66b715e4bdf (127.10.89.126:36189), d6bb4436c3c94a9c89ef988fe1e68010 (127.10.89.124:42407)
14:02:52.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.958413 10897 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 43052a37ca8648629532aac191629eb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
14:02:52.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.958727 10897 raft_consensus.cc:493] T 00000000000000000000000000000000 P 43052a37ca8648629532aac191629eb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
14:02:52.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.958995 10897 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 43052a37ca8648629532aac191629eb2 [term 0 FOLLOWER]: Advancing to term 1
14:02:52.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.960192 10897 raft_consensus.cc:515] T 00000000000000000000000000000000 P 43052a37ca8648629532aac191629eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155b8f49c1ea408b8f15e66b715e4bdf" member_type: VOTER last_known_addr { host: "127.10.89.126" port: 36189 } } peers { permanent_uuid: "43052a37ca8648629532aac191629eb2" member_type: VOTER last_known_addr { host: "127.10.89.125" port: 42399 } } peers { permanent_uuid: "d6bb4436c3c94a9c89ef988fe1e68010" member_type: VOTER last_known_addr { host: "127.10.89.124" port: 42407 } }
14:02:52.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.960752 10658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "43052a37ca8648629532aac191629eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155b8f49c1ea408b8f15e66b715e4bdf"
14:02:52.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.960836 10658 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 155b8f49c1ea408b8f15e66b715e4bdf [term 0 FOLLOWER]: Advancing to term 1
14:02:52.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.961695 10658 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 155b8f49c1ea408b8f15e66b715e4bdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43052a37ca8648629532aac191629eb2 in term 1.
14:02:52.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.961871 10719 leader_election.cc:304] T 00000000000000000000000000000000 P 43052a37ca8648629532aac191629eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155b8f49c1ea408b8f15e66b715e4bdf, 43052a37ca8648629532aac191629eb2; no voters: 
14:02:52.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.963994 10828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "43052a37ca8648629532aac191629eb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6bb4436c3c94a9c89ef988fe1e68010"
14:02:52.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.964108 10828 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d6bb4436c3c94a9c89ef988fe1e68010 [term 0 FOLLOWER]: Advancing to term 1
14:02:52.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.965346 10828 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d6bb4436c3c94a9c89ef988fe1e68010 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43052a37ca8648629532aac191629eb2 in term 1.
14:02:52.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.967952 10897 leader_election.cc:290] T 00000000000000000000000000000000 P 43052a37ca8648629532aac191629eb2 [CANDIDATE]: Term 1 election: Requested vote from peers 155b8f49c1ea408b8f15e66b715e4bdf (127.10.89.126:36189), d6bb4436c3c94a9c89ef988fe1e68010 (127.10.89.124:42407)
14:02:52.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.968899 10897 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 43052a37ca8648629532aac191629eb2 [term 1 FOLLOWER]: Leader election won for term 1
14:02:52.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.969234 10897 raft_consensus.cc:697] T 00000000000000000000000000000000 P 43052a37ca8648629532aac191629eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 43052a37ca8648629532aac191629eb2, State: Running, Role: LEADER
14:02:52.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.971079 10897 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43052a37ca8648629532aac191629eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155b8f49c1ea408b8f15e66b715e4bdf" member_type: VOTER last_known_addr { host: "127.10.89.126" port: 36189 } } peers { permanent_uuid: "43052a37ca8648629532aac191629eb2" member_type: VOTER last_known_addr { host: "127.10.89.125" port: 42399 } } peers { permanent_uuid: "d6bb4436c3c94a9c89ef988fe1e68010" member_type: VOTER last_known_addr { host: "127.10.89.124" port: 42407 } }
14:02:52.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.977722 10897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43052a37ca8648629532aac191629eb2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43052a37ca8648629532aac191629eb2. Latest consensus state: current_term: 1 leader_uuid: "43052a37ca8648629532aac191629eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155b8f49c1ea408b8f15e66b715e4bdf" member_type: VOTER last_known_addr { host: "127.10.89.126" port: 36189 } } peers { permanent_uuid: "43052a37ca8648629532aac191629eb2" member_type: VOTER last_known_addr { host: "127.10.89.125" port: 42399 } } peers { permanent_uuid: "d6bb4436c3c94a9c89ef988fe1e68010" member_type: VOTER last_known_addr { host: "127.10.89.124" port: 42407 } } }
14:02:52.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.978399 10897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43052a37ca8648629532aac191629eb2 [sys.catalog]: This master's current role is: LEADER
14:02:52.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.978976 10912 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
14:02:52.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.979609 10912 catalog_manager.cc:1520] Initializing Kudu cluster ID...
14:02:52.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:52.984752 10632 tablet.cc:2406] T 00000000000000000000000000000000 P 155b8f49c1ea408b8f15e66b715e4bdf: Can't schedule compaction. Clean time has not been advanced past its initial value.
14:02:52.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.986027 10828 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d6bb4436c3c94a9c89ef988fe1e68010 [term 1 FOLLOWER]: Refusing update from remote peer 43052a37ca8648629532aac191629eb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
14:02:52.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.986312 10658 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 155b8f49c1ea408b8f15e66b715e4bdf [term 1 FOLLOWER]: Refusing update from remote peer 43052a37ca8648629532aac191629eb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
14:02:52.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.986663 10910 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 43052a37ca8648629532aac191629eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155b8f49c1ea408b8f15e66b715e4bdf" member_type: VOTER last_known_addr { host: "127.10.89.126" port: 36189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
14:02:52.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.987020 10910 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 43052a37ca8648629532aac191629eb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6bb4436c3c94a9c89ef988fe1e68010" member_type: VOTER last_known_addr { host: "127.10.89.124" port: 42407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
14:02:52.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.988194 10882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 155b8f49c1ea408b8f15e66b715e4bdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43052a37ca8648629532aac191629eb2. Latest consensus state: current_term: 1 leader_uuid: "43052a37ca8648629532aac191629eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155b8f49c1ea408b8f15e66b715e4bdf" member_type: VOTER last_known_addr { host: "127.10.89.126" port: 36189 } } peers { permanent_uuid: "43052a37ca8648629532aac191629eb2" member_type: VOTER last_known_addr { host: "127.10.89.125" port: 42399 } } peers { permanent_uuid: "d6bb4436c3c94a9c89ef988fe1e68010" member_type: VOTER last_known_addr { host: "127.10.89.124" port: 42407 } } }
14:02:52.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.988283 10882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 155b8f49c1ea408b8f15e66b715e4bdf [sys.catalog]: This master's current role is: FOLLOWER
14:02:52.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.989075 10866 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6bb4436c3c94a9c89ef988fe1e68010 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43052a37ca8648629532aac191629eb2. Latest consensus state: current_term: 1 leader_uuid: "43052a37ca8648629532aac191629eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155b8f49c1ea408b8f15e66b715e4bdf" member_type: VOTER last_known_addr { host: "127.10.89.126" port: 36189 } } peers { permanent_uuid: "43052a37ca8648629532aac191629eb2" member_type: VOTER last_known_addr { host: "127.10.89.125" port: 42399 } } peers { permanent_uuid: "d6bb4436c3c94a9c89ef988fe1e68010" member_type: VOTER last_known_addr { host: "127.10.89.124" port: 42407 } } }
14:02:52.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.989166 10866 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6bb4436c3c94a9c89ef988fe1e68010 [sys.catalog]: This master's current role is: FOLLOWER
14:02:52.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.990754 10910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43052a37ca8648629532aac191629eb2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "43052a37ca8648629532aac191629eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155b8f49c1ea408b8f15e66b715e4bdf" member_type: VOTER last_known_addr { host: "127.10.89.126" port: 36189 } } peers { permanent_uuid: "43052a37ca8648629532aac191629eb2" member_type: VOTER last_known_addr { host: "127.10.89.125" port: 42399 } } peers { permanent_uuid: "d6bb4436c3c94a9c89ef988fe1e68010" member_type: VOTER last_known_addr { host: "127.10.89.124" port: 42407 } } }
14:02:52.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.990846 10910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43052a37ca8648629532aac191629eb2 [sys.catalog]: This master's current role is: LEADER
14:02:52.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.992359 10866 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6bb4436c3c94a9c89ef988fe1e68010 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "43052a37ca8648629532aac191629eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155b8f49c1ea408b8f15e66b715e4bdf" member_type: VOTER last_known_addr { host: "127.10.89.126" port: 36189 } } peers { permanent_uuid: "43052a37ca8648629532aac191629eb2" member_type: VOTER last_known_addr { host: "127.10.89.125" port: 42399 } } peers { permanent_uuid: "d6bb4436c3c94a9c89ef988fe1e68010" member_type: VOTER last_known_addr { host: "127.10.89.124" port: 42407 } } }
14:02:52.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.992437 10866 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6bb4436c3c94a9c89ef988fe1e68010 [sys.catalog]: This master's current role is: FOLLOWER
14:02:52.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.993657 10913 mvcc.cc:204] Tried to move back new op lower bound from 7317327761329360896 to 7317327761291886592. Current Snapshot: MvccSnapshot[applied={T|T < 7317327761329360896}]
14:02:52.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.994684 10918 mvcc.cc:204] Tried to move back new op lower bound from 7317327761329360896 to 7317327761291886592. Current Snapshot: MvccSnapshot[applied={T|T < 7317327761329360896}]
14:02:52.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.995471 10912 catalog_manager.cc:1383] Generated new cluster ID: 3921337f856a445fb2a8872ec11fb2f8
14:02:52.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.995519 10912 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
14:02:52.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.996068 10897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43052a37ca8648629532aac191629eb2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "43052a37ca8648629532aac191629eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155b8f49c1ea408b8f15e66b715e4bdf" member_type: VOTER last_known_addr { host: "127.10.89.126" port: 36189 } } peers { permanent_uuid: "43052a37ca8648629532aac191629eb2" member_type: VOTER last_known_addr { host: "127.10.89.125" port: 42399 } } peers { permanent_uuid: "d6bb4436c3c94a9c89ef988fe1e68010" member_type: VOTER last_known_addr { host: "127.10.89.124" port: 42407 } } }
14:02:52.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.996150 10897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43052a37ca8648629532aac191629eb2 [sys.catalog]: This master's current role is: LEADER
14:02:52.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.996596 10882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 155b8f49c1ea408b8f15e66b715e4bdf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "43052a37ca8648629532aac191629eb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155b8f49c1ea408b8f15e66b715e4bdf" member_type: VOTER last_known_addr { host: "127.10.89.126" port: 36189 } } peers { permanent_uuid: "43052a37ca8648629532aac191629eb2" member_type: VOTER last_known_addr { host: "127.10.89.125" port: 42399 } } peers { permanent_uuid: "d6bb4436c3c94a9c89ef988fe1e68010" member_type: VOTER last_known_addr { host: "127.10.89.124" port: 42407 } } }
14:02:52.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.996681 10882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 155b8f49c1ea408b8f15e66b715e4bdf [sys.catalog]: This master's current role is: FOLLOWER
14:02:52.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:52.998381 10914 mvcc.cc:204] Tried to move back new op lower bound from 7317327761329360896 to 7317327761291886592. Current Snapshot: MvccSnapshot[applied={T|T < 7317327761329360896}]
14:02:53.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.015853 10912 catalog_manager.cc:1406] Generated new certificate authority record
14:02:53.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.017132 10912 catalog_manager.cc:1540] Loading token signing keys...
14:02:53.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.027088 10912 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 43052a37ca8648629532aac191629eb2: Generated new TSK 0
14:02:53.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.027941 10912 catalog_manager.cc:1550] Initializing in-progress tserver states...
14:02:53.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.038389 10597 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskhBFXjG/build/release/bin/kudu
14:02:53.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskhBFXjG/build/release/bin/kudu
14:02:53.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/wal
14:02:53.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/data
14:02:53.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:02:53.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:02:53.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:02:53.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:02:53.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:02:53.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:02:53.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/logs
14:02:53.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/data/info.pb
14:02:53.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:02:53.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:02:53.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:02:53.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:02:53.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:02:53.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:02:53.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:02:53.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:02:53.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:02:53.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:02:53.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.89.65:0
14:02:53.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.89.65
14:02:53.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.89.65
14:02:53.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:02:53.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.89.126:36189,127.10.89.125:42399,127.10.89.124:42407
14:02:53.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.89.84:33487
14:02:53.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:02:53.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:02:53.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:02:53.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:53.124929 10924 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:02:53.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:53.125141 10924 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:02:53.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:53.125175 10924 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:02:53.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:53.127012 10924 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:02:53.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:53.127117 10924 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.89.65
14:02:53.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.128932 10924 tablet_server_runner.cc:78] Tablet server non-default flags:
14:02:53.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/logs
14:02:53.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:02:53.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:02:53.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:02:53.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.89.84:33487
14:02:53.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:02:53.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:02:53.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/data
14:02:53.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/wal
14:02:53.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:02:53.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:02:53.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.89.65:0
14:02:53.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:02:53.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:02:53.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:02:53.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/data/info.pb
14:02:53.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.89.65
14:02:53.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:02:53.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.89.126:36189,127.10.89.125:42399,127.10.89.124:42407
14:02:53.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:02:53.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.10924
14:02:53.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:02:53.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:02:53.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:02:53.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:02:53.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.89.65
14:02:53.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:02:53.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:02:53.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:02:53.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 4bcd631c449d3c6649730ca6e07660c8168f0d44
14:02:53.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
14:02:53.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Aug 2026 13:43:13 UTC on e7f111948823
14:02:53.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 13289
14:02:53.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.129240 10924 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:02:53.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.129541 10924 file_cache.cc:504] Constructed file cache file cache with capacity 419430
14:02:53.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:53.132700 10930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:02:53.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:53.132750 10931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:02:53.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:53.132723 10933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:02:53.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.133432 10924 server_base.cc:1061] running on GCE node
14:02:53.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.133638 10924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:02:53.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.133879 10924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:02:53.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.135067 10924 hybrid_clock.cc:648] HybridClock initialized: now 1786456973135033 us; error 49 us; skew 500 ppm
14:02:53.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.136497 10924 webserver.cc:533] Webserver started at http://127.10.89.65:43475/ using document root <none> and password file <none>
14:02:53.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.136730 10924 fs_manager.cc:362] Metadata directory not provided
14:02:53.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.136792 10924 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:02:53.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.136914 10924 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
14:02:53.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.138058 10924 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/data/instance:
14:02:53.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d8b415ec6a9a4671b34e29775c855962"
14:02:53.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 14:02:53 on dist-test-slave-g350"
14:02:53.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.138455 10924 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/wal/instance:
14:02:53.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d8b415ec6a9a4671b34e29775c855962"
14:02:53.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 14:02:53 on dist-test-slave-g350"
14:02:53.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.140039 10924 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.002s	sys 0.000s
14:02:53.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.140862 10939 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:02:53.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.141055 10924 fs_manager.cc:730] Time spent opening block manager: real 0.001s	user 0.001s	sys 0.000s
14:02:53.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.141132 10924 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/data,/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/wal
14:02:53.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d8b415ec6a9a4671b34e29775c855962"
14:02:53.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 14:02:53 on dist-test-slave-g350"
14:02:53.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.141186 10924 fs_report.cc:389] FS layout report
14:02:53.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:02:53.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/wal
14:02:53.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/wal
14:02:53.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/data/data
14:02:53.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:02:53.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:02:53.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:02:53.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:02:53.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:02:53.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:02:53.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:02:53.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:02:53.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:02:53.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:02:53.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:02:53.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.152786 10924 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:02:53.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.153152 10924 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:02:53.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.153281 10924 kserver.cc:163] Server-wide thread pool size limit: 3276
14:02:53.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.153548 10924 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:02:53.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.153978 10924 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
14:02:53.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.154024 10924 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
14:02:53.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.154053 10924 ts_tablet_manager.cc:616] Registered 0 tablets
14:02:53.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.154071 10924 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
14:02:53.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.161827 10924 rpc_server.cc:307] RPC server started. Bound to: 127.10.89.65:36825
14:02:53.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.161911 11074 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.89.65:36825 every 8 connection(s)
14:02:53.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.162256 10924 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/data/info.pb
14:02:53.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.164938 10597 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskhBFXjG/build/release/bin/kudu as pid 10924
14:02:53.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.165043 10597 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/wal/instance
14:02:53.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.166934 10597 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskhBFXjG/build/release/bin/kudu
14:02:53.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskhBFXjG/build/release/bin/kudu
14:02:53.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/wal
14:02:53.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/data
14:02:53.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:02:53.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:02:53.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:02:53.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:02:53.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:02:53.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:02:53.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/logs
14:02:53.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/data/info.pb
14:02:53.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:02:53.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:02:53.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:02:53.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:02:53.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:02:53.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:02:53.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:02:53.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:02:53.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:02:53.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:02:53.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.89.66:0
14:02:53.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.89.66
14:02:53.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.89.66
14:02:53.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:02:53.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.89.126:36189,127.10.89.125:42399,127.10.89.124:42407
14:02:53.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.89.84:33487
14:02:53.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:02:53.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:02:53.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:02:53.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.169612 11075 heartbeater.cc:344] Connected to a master server at 127.10.89.124:42407
14:02:53.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.169718 11075 heartbeater.cc:461] Registering TS with master...
14:02:53.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.169955 11075 heartbeater.cc:507] Master 127.10.89.124:42407 requested a full tablet report, sending...
14:02:53.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.170691 11076 heartbeater.cc:344] Connected to a master server at 127.10.89.126:36189
14:02:53.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.170747 11076 heartbeater.cc:461] Registering TS with master...
14:02:53.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.170797 10814 ts_manager.cc:194] Registered new tserver with Master: d8b415ec6a9a4671b34e29775c855962 (127.10.89.65:36825)
14:02:53.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.170873 11076 heartbeater.cc:507] Master 127.10.89.126:36189 requested a full tablet report, sending...
14:02:53.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.171300 10644 ts_manager.cc:194] Registered new tserver with Master: d8b415ec6a9a4671b34e29775c855962 (127.10.89.65:36825)
14:02:53.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.171418 11077 heartbeater.cc:344] Connected to a master server at 127.10.89.125:42399
14:02:53.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.171473 11077 heartbeater.cc:461] Registering TS with master...
14:02:53.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.171563 11077 heartbeater.cc:507] Master 127.10.89.125:42399 requested a full tablet report, sending...
14:02:53.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.171938 10732 ts_manager.cc:194] Registered new tserver with Master: d8b415ec6a9a4671b34e29775c855962 (127.10.89.65:36825)
14:02:53.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.172917 10732 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.10.89.65:36255
14:02:53.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:53.252874 11082 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:02:53.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:53.253103 11082 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:02:53.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:53.253162 11082 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:02:53.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:53.255177 11082 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:02:53.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:53.255262 11082 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.89.66
14:02:53.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.257021 11082 tablet_server_runner.cc:78] Tablet server non-default flags:
14:02:53.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/logs
14:02:53.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:02:53.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:02:53.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:02:53.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.89.84:33487
14:02:53.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:02:53.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:02:53.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/data
14:02:53.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/wal
14:02:53.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:02:53.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:02:53.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.89.66:0
14:02:53.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:02:53.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:02:53.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:02:53.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/data/info.pb
14:02:53.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.89.66
14:02:53.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:02:53.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.89.126:36189,127.10.89.125:42399,127.10.89.124:42407
14:02:53.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:02:53.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.11082
14:02:53.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:02:53.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:02:53.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:02:53.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:02:53.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.89.66
14:02:53.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:02:53.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:02:53.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:02:53.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 4bcd631c449d3c6649730ca6e07660c8168f0d44
14:02:53.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
14:02:53.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Aug 2026 13:43:13 UTC on e7f111948823
14:02:53.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 13289
14:02:53.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.257313 11082 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:02:53.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.257601 11082 file_cache.cc:504] Constructed file cache file cache with capacity 419430
14:02:53.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:53.260725 11088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:02:53.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:53.260715 11087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:02:53.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:53.260713 11090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:02:53.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.261058 11082 server_base.cc:1061] running on GCE node
14:02:53.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.261229 11082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:02:53.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.261479 11082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:02:53.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.262650 11082 hybrid_clock.cc:648] HybridClock initialized: now 1786456973262627 us; error 33 us; skew 500 ppm
14:02:53.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.263875 11082 webserver.cc:533] Webserver started at http://127.10.89.66:39225/ using document root <none> and password file <none>
14:02:53.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.264111 11082 fs_manager.cc:362] Metadata directory not provided
14:02:53.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.264170 11082 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:02:53.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.264302 11082 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
14:02:53.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.265232 11082 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/data/instance:
14:02:53.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "7bf0930626894cff94471c02ef2b9196"
14:02:53.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 14:02:53 on dist-test-slave-g350"
14:02:53.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.265641 11082 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/wal/instance:
14:02:53.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "7bf0930626894cff94471c02ef2b9196"
14:02:53.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 14:02:53 on dist-test-slave-g350"
14:02:53.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.266999 11082 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.001s	sys 0.000s
14:02:53.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.267751 11096 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:02:53.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.267953 11082 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.001s	sys 0.000s
14:02:53.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.268023 11082 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/data,/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/wal
14:02:53.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "7bf0930626894cff94471c02ef2b9196"
14:02:53.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 14:02:53 on dist-test-slave-g350"
14:02:53.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.268085 11082 fs_report.cc:389] FS layout report
14:02:53.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:02:53.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/wal
14:02:53.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/wal
14:02:53.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/data/data
14:02:53.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:02:53.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:02:53.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:02:53.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:02:53.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:02:53.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:02:53.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:02:53.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:02:53.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:02:53.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:02:53.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:02:53.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.290869 11082 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:02:53.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.291175 11082 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:02:53.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.291265 11082 kserver.cc:163] Server-wide thread pool size limit: 3276
14:02:53.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.291463 11082 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:02:53.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.291750 11082 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
14:02:53.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.291774 11082 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
14:02:53.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.291796 11082 ts_tablet_manager.cc:616] Registered 0 tablets
14:02:53.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.291819 11082 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
14:02:53.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.299269 11082 rpc_server.cc:307] RPC server started. Bound to: 127.10.89.66:34685
14:02:53.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.299446 11226 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.89.66:34685 every 8 connection(s)
14:02:53.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.299652 11082 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/data/info.pb
14:02:53.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.305306 11228 heartbeater.cc:344] Connected to a master server at 127.10.89.126:36189
14:02:53.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.305373 11229 heartbeater.cc:344] Connected to a master server at 127.10.89.125:42399
14:02:53.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.305326 11227 heartbeater.cc:344] Connected to a master server at 127.10.89.124:42407
14:02:53.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.305433 11228 heartbeater.cc:461] Registering TS with master...
14:02:53.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.305437 11229 heartbeater.cc:461] Registering TS with master...
14:02:53.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.305475 11227 heartbeater.cc:461] Registering TS with master...
14:02:53.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.305696 11227 heartbeater.cc:507] Master 127.10.89.124:42407 requested a full tablet report, sending...
14:02:53.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.305694 11229 heartbeater.cc:507] Master 127.10.89.125:42399 requested a full tablet report, sending...
14:02:53.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.305694 11228 heartbeater.cc:507] Master 127.10.89.126:36189 requested a full tablet report, sending...
14:02:53.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.306180 10644 ts_manager.cc:194] Registered new tserver with Master: 7bf0930626894cff94471c02ef2b9196 (127.10.89.66:34685)
14:02:53.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.306227 10814 ts_manager.cc:194] Registered new tserver with Master: 7bf0930626894cff94471c02ef2b9196 (127.10.89.66:34685)
14:02:53.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.306348 10597 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskhBFXjG/build/release/bin/kudu as pid 11082
14:02:53.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.306421 10597 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/wal/instance
14:02:53.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.307013 10731 ts_manager.cc:194] Registered new tserver with Master: 7bf0930626894cff94471c02ef2b9196 (127.10.89.66:34685)
14:02:53.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.307519 10731 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.10.89.66:58953
14:02:53.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.308290 10597 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskhBFXjG/build/release/bin/kudu
14:02:53.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskhBFXjG/build/release/bin/kudu
14:02:53.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/wal
14:02:53.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/data
14:02:53.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:02:53.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:02:53.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:02:53.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:02:53.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:02:53.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:02:53.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/logs
14:02:53.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/data/info.pb
14:02:53.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:02:53.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:02:53.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:02:53.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:02:53.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:02:53.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:02:53.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:02:53.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:02:53.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:02:53.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:02:53.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.89.67:0
14:02:53.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.89.67
14:02:53.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.89.67
14:02:53.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:02:53.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.89.126:36189,127.10.89.125:42399,127.10.89.124:42407
14:02:53.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.89.84:33487
14:02:53.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:02:53.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:02:53.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:02:53.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:53.390944 11234 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:02:53.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:53.391155 11234 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:02:53.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:53.391186 11234 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:02:53.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:53.393010 11234 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:02:53.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:53.393105 11234 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.89.67
14:02:53.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.394876 11234 tablet_server_runner.cc:78] Tablet server non-default flags:
14:02:53.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/logs
14:02:53.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:02:53.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:02:53.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:02:53.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.89.84:33487
14:02:53.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:02:53.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:02:53.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/data
14:02:53.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/wal
14:02:53.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:02:53.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:02:53.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.89.67:0
14:02:53.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:02:53.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:02:53.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:02:53.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/data/info.pb
14:02:53.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.89.67
14:02:53.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:02:53.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.89.126:36189,127.10.89.125:42399,127.10.89.124:42407
14:02:53.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:02:53.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.11234
14:02:53.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:02:53.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:02:53.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:02:53.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:02:53.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.89.67
14:02:53.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:02:53.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:02:53.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:02:53.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 4bcd631c449d3c6649730ca6e07660c8168f0d44
14:02:53.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
14:02:53.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Aug 2026 13:43:13 UTC on e7f111948823
14:02:53.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 13289
14:02:53.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.395161 11234 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:02:53.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.395465 11234 file_cache.cc:504] Constructed file cache file cache with capacity 419430
14:02:53.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:53.398752 11242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:02:53.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:53.398764 11240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:02:53.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:53.398775 11239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:02:53.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.399077 11234 server_base.cc:1061] running on GCE node
14:02:53.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.399338 11234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:02:53.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.399570 11234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:02:53.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.400743 11234 hybrid_clock.cc:648] HybridClock initialized: now 1786456973400716 us; error 44 us; skew 500 ppm
14:02:53.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.401891 11234 webserver.cc:533] Webserver started at http://127.10.89.67:34339/ using document root <none> and password file <none>
14:02:53.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.402123 11234 fs_manager.cc:362] Metadata directory not provided
14:02:53.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.402264 11234 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:02:53.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.402403 11234 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
14:02:53.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.403301 11234 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/data/instance:
14:02:53.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "751f3be0b90f4a1d86af88ca13d42016"
14:02:53.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 14:02:53 on dist-test-slave-g350"
14:02:53.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.403671 11234 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/wal/instance:
14:02:53.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "751f3be0b90f4a1d86af88ca13d42016"
14:02:53.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 14:02:53 on dist-test-slave-g350"
14:02:53.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.405092 11234 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.000s	sys 0.003s
14:02:53.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.405910 11248 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:02:53.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.406129 11234 fs_manager.cc:730] Time spent opening block manager: real 0.001s	user 0.000s	sys 0.001s
14:02:53.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.406198 11234 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/data,/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/wal
14:02:53.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "751f3be0b90f4a1d86af88ca13d42016"
14:02:53.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 14:02:53 on dist-test-slave-g350"
14:02:53.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.406261 11234 fs_report.cc:389] FS layout report
14:02:53.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:02:53.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/wal
14:02:53.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/wal
14:02:53.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/data/data
14:02:53.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:02:53.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:02:53.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:02:53.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:02:53.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:02:53.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:02:53.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:02:53.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:02:53.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:02:53.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:02:53.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:02:53.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.422420 11234 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:02:53.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.422839 11234 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:02:53.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.422981 11234 kserver.cc:163] Server-wide thread pool size limit: 3276
14:02:53.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.423257 11234 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:02:53.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.423727 11234 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
14:02:53.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.423794 11234 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
14:02:53.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.423830 11234 ts_tablet_manager.cc:616] Registered 0 tablets
14:02:53.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.423849 11234 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
14:02:53.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.430509 11234 rpc_server.cc:307] RPC server started. Bound to: 127.10.89.67:46723
14:02:53.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.430555 11379 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.89.67:46723 every 8 connection(s)
14:02:53.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.430883 11234 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/data/info.pb
14:02:53.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.433830 10597 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskhBFXjG/build/release/bin/kudu as pid 11234
14:02:53.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.433941 10597 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/wal/instance
14:02:53.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.437228 11380 heartbeater.cc:344] Connected to a master server at 127.10.89.124:42407
14:02:53.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.437260 11381 heartbeater.cc:344] Connected to a master server at 127.10.89.126:36189
14:02:53.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.437362 11381 heartbeater.cc:461] Registering TS with master...
14:02:53.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.437364 11380 heartbeater.cc:461] Registering TS with master...
14:02:53.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.437611 11380 heartbeater.cc:507] Master 127.10.89.124:42407 requested a full tablet report, sending...
14:02:53.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.437718 11381 heartbeater.cc:507] Master 127.10.89.126:36189 requested a full tablet report, sending...
14:02:53.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.438118 10814 ts_manager.cc:194] Registered new tserver with Master: 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723)
14:02:53.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.438118 10644 ts_manager.cc:194] Registered new tserver with Master: 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723)
14:02:53.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.438426 11382 heartbeater.cc:344] Connected to a master server at 127.10.89.125:42399
14:02:53.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.438494 11382 heartbeater.cc:461] Registering TS with master...
14:02:53.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.438591 11382 heartbeater.cc:507] Master 127.10.89.125:42399 requested a full tablet report, sending...
14:02:53.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.438944 10732 ts_manager.cc:194] Registered new tserver with Master: 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723)
14:02:53.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.439383 10732 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.10.89.67:45121
14:02:53.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.449234 10597 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
14:02:53.450 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
14:02:53.451 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_masters {
}

14:02:53.460 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_masters {
  masters {
    id {
      type: MASTER
      index: 0
    }
    bound_rpc_address {
      host: "127.10.89.126"
      port: 36189
    }
    bound_http_address {
      host: "127.10.89.126"
      port: 39697
    }
  }
  masters {
    id {
      type: MASTER
      index: 1
    }
    bound_rpc_address {
      host: "127.10.89.125"
      port: 42399
    }
    bound_http_address {
      host: "127.10.89.125"
      port: 39057
    }
  }
  masters {
    id {
      type: MASTER
      index: 2
    }
    bound_rpc_address {
      host: "127.10.89.124"
      port: 42407
    }
    bound_http_address {
      host: "127.10.89.124"
      port: 38271
    }
  }
}

14:02:53.463 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_tservers {
}

14:02:53.465 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_tservers {
  tservers {
    id {
      type: TSERVER
      index: 0
    }
    bound_rpc_address {
      host: "127.10.89.65"
      port: 36825
    }
    bound_http_address {
      host: "127.10.89.65"
      port: 43475
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 1
    }
    bound_rpc_address {
      host: "127.10.89.66"
      port: 34685
    }
    bound_http_address {
      host: "127.10.89.66"
      port: 39225
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 2
    }
    bound_rpc_address {
      host: "127.10.89.67"
      port: 46723
    }
    bound_http_address {
      host: "127.10.89.67"
      port: 34339
    }
  }
}

14:02:53.466 [INFO - main] (KuduTestHarness.java:157) Creating a new Kudu client...
14:02:53.640 [DEBUG - main] (CompositeStrategy.java:59) Unable to load modern Subject methods, relying only on legacy methods
java.lang.NoSuchMethodException: javax.security.auth.Subject.current()
	at java.base/java.lang.Class.getDeclaredMethod(Class.java:2677)
	at org.apache.kudu.client.internals.ModernStrategy.<init>(ModernStrategy.java:51)
	at org.apache.kudu.client.internals.CompositeStrategy.<init>(CompositeStrategy.java:52)
	at org.apache.kudu.client.internals.CompositeStrategy.<clinit>(CompositeStrategy.java:41)
	at org.apache.kudu.client.internals.SecurityManagerCompatibility.get(SecurityManagerCompatibility.java:43)
	at org.apache.kudu.client.SecurityContext.setupSubject(SecurityContext.java:155)
	at org.apache.kudu.client.SecurityContext.<init>(SecurityContext.java:138)
	at org.apache.kudu.client.AsyncKuduClient.<init>(AsyncKuduClient.java:431)
	at org.apache.kudu.client.AsyncKuduClient$AsyncKuduClientBuilder.build(AsyncKuduClient.java:3109)
	at org.apache.kudu.test.KuduTestHarness.before(KuduTestHarness.java:161)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:50)
	at org.apache.kudu.test.junit.RetryRule$RetryStatement.doOneAttemptAndReport(RetryRule.java:184)
	at org.apache.kudu.test.junit.RetryRule$RetryStatement.evaluate(RetryRule.java:244)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:27)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
	at org.junit.runner.JUnitCore.runMain(JUnitCore.java:77)
	at org.junit.runner.JUnitCore.main(JUnitCore.java:36)
14:02:53.706 [DEBUG - main] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 
14:02:53.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.780769 10879 catalog_manager.cc:1295] Loaded cluster ID: 3921337f856a445fb2a8872ec11fb2f8
14:02:53.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.780865 10879 catalog_manager.cc:1588] T 00000000000000000000000000000000 P d6bb4436c3c94a9c89ef988fe1e68010: loading cluster ID for follower catalog manager: success
14:02:53.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.782232 10879 catalog_manager.cc:1610] T 00000000000000000000000000000000 P d6bb4436c3c94a9c89ef988fe1e68010: acquiring CA information for follower catalog manager: success
14:02:53.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.782586 10879 catalog_manager.cc:1638] T 00000000000000000000000000000000 P d6bb4436c3c94a9c89ef988fe1e68010: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
14:02:53.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.841697 10894 catalog_manager.cc:1295] Loaded cluster ID: 3921337f856a445fb2a8872ec11fb2f8
14:02:53.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.841782 10894 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 155b8f49c1ea408b8f15e66b715e4bdf: loading cluster ID for follower catalog manager: success
14:02:53.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.843003 10894 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 155b8f49c1ea408b8f15e66b715e4bdf: acquiring CA information for follower catalog manager: success
14:02:53.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:53.843339 10894 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 155b8f49c1ea408b8f15e66b715e4bdf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
14:02:53.891 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.10.89.124:42407(127.10.89.124:42407)] connecting to peer
14:02:53.955 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.10.89.125:42399(127.10.89.125:42399)] connecting to peer
14:02:53.956 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.10.89.126:36189(127.10.89.126:36189)] connecting to peer
14:02:53.974 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.10.89.124:42407(127.10.89.124:42407)] Successfully connected to peer
14:02:53.974 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.10.89.125:42399(127.10.89.125:42399)] Successfully connected to peer
14:02:53.974 [DEBUG - kudu-nio-2] (Connection.java:757) [peer master-127.10.89.126:36189(127.10.89.126:36189)] Successfully connected to peer
14:02:54.076 [DEBUG - kudu-nio-0] (Negotiator.java:528) SASL mechanism PLAIN chosen for peer 127.10.89.124
14:02:54.076 [DEBUG - kudu-nio-1] (Negotiator.java:528) SASL mechanism PLAIN chosen for peer 127.10.89.125
14:02:54.078 [DEBUG - kudu-nio-2] (Negotiator.java:528) SASL mechanism PLAIN chosen for peer 127.10.89.126
14:02:54.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.174201 11077 heartbeater.cc:499] Master 127.10.89.125:42399 was elected leader, sending a full tablet report...
14:02:54.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.308326 11229 heartbeater.cc:499] Master 127.10.89.125:42399 was elected leader, sending a full tablet report...
14:02:54.314 [DEBUG - kudu-nio-2] (Negotiator.java:918) Authenticated connection [id: 0xacd2ce0a, L:/127.0.0.1:51514 - R:/127.10.89.126:36189] using SASL/PLAIN
14:02:54.316 [DEBUG - kudu-nio-1] (Negotiator.java:918) Authenticated connection [id: 0xbf4efb2b, L:/127.0.0.1:39038 - R:/127.10.89.125:42399] using SASL/PLAIN
14:02:54.316 [DEBUG - kudu-nio-0] (Negotiator.java:918) Authenticated connection [id: 0xc44a8949, L:/127.0.0.1:60376 - R:/127.10.89.124:42407] using SASL/PLAIN
14:02:54.354 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet Kudu Master for table 'Kudu Master' with partition [<start>, <end>)
14:02:54.356 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=CreateTable, tablet=null, attempt=1, TimeoutTracker(timeout=50000, elapsed=512), Traces: [0ms] refreshing cache from master, [30ms] Sub RPC ConnectToMaster: sending RPC to server master-127.10.89.124:42407, [106ms] Sub RPC ConnectToMaster: sending RPC to server master-127.10.89.125:42399, [107ms] Sub RPC ConnectToMaster: sending RPC to server master-127.10.89.126:36189, [480ms] Sub RPC ConnectToMaster: received response from server master-127.10.89.126:36189: OK, [480ms] Sub RPC ConnectToMaster: received response from server master-127.10.89.124:42407: OK, [481ms] Sub RPC ConnectToMaster: received response from server master-127.10.89.125:42399: OK, deferred=null) after lookup
14:02:54.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.401834 10732 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
14:02:54.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) name: "org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823"
14:02:54.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) schema {
14:02:54.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
14:02:54.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "key"
14:02:54.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
14:02:54.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: true
14:02:54.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
14:02:54.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
14:02:54.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
14:02:54.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
14:02:54.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
14:02:54.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
14:02:54.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column1_i"
14:02:54.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
14:02:54.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
14:02:54.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
14:02:54.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
14:02:54.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
14:02:54.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
14:02:54.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
14:02:54.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
14:02:54.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column2_i"
14:02:54.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
14:02:54.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
14:02:54.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
14:02:54.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
14:02:54.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
14:02:54.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
14:02:54.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
14:02:54.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
14:02:54.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column3_s"
14:02:54.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: STRING
14:02:54.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
14:02:54.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: true
14:02:54.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     encoding: DICT_ENCODING
14:02:54.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     compression: LZ4
14:02:54.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 4096
14:02:54.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
14:02:54.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
14:02:54.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
14:02:54.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
14:02:54.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column4_b"
14:02:54.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: BOOL
14:02:54.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
14:02:54.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
14:02:54.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
14:02:54.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
14:02:54.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
14:02:54.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
14:02:54.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
14:02:54.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) partition_schema {
14:02:54.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   range_schema {
14:02:54.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     columns {
14:02:54.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)       name: "key"
14:02:54.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     }
14:02:54.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
14:02:54.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
14:02:54.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:54.403434 10732 catalog_manager.cc:7099] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
14:02:54.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.416592 10994 tablet_service.cc:1511] Processing CreateTablet for tablet 28da754494014b9f85434e4ebbbef9d7 (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823 [id=7eede73d066f4e5491d4fa0ef0f8a051]), partition=RANGE (key) PARTITION UNBOUNDED
14:02:54.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.416945 10994 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28da754494014b9f85434e4ebbbef9d7. 1 dirs total, 0 dirs full, 0 dirs failed
14:02:54.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.417871 11151 tablet_service.cc:1511] Processing CreateTablet for tablet 28da754494014b9f85434e4ebbbef9d7 (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823 [id=7eede73d066f4e5491d4fa0ef0f8a051]), partition=RANGE (key) PARTITION UNBOUNDED
14:02:54.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.418152 11151 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28da754494014b9f85434e4ebbbef9d7. 1 dirs total, 0 dirs full, 0 dirs failed
14:02:54.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.420152 11421 tablet_bootstrap.cc:492] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962: Bootstrap starting.
14:02:54.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.420343 11299 tablet_service.cc:1511] Processing CreateTablet for tablet 28da754494014b9f85434e4ebbbef9d7 (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823 [id=7eede73d066f4e5491d4fa0ef0f8a051]), partition=RANGE (key) PARTITION UNBOUNDED
14:02:54.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.420639 11299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28da754494014b9f85434e4ebbbef9d7. 1 dirs total, 0 dirs full, 0 dirs failed
14:02:54.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.420851 11421 tablet_bootstrap.cc:654] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962: Neither blocks nor log segments found. Creating new log.
14:02:54.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.421142 11421 log.cc:826] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962: Log is configured to *not* fsync() on all Append() calls
14:02:54.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.421761 11422 tablet_bootstrap.cc:492] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196: Bootstrap starting.
14:02:54.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.423177 11422 tablet_bootstrap.cc:654] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196: Neither blocks nor log segments found. Creating new log.
14:02:54.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.423441 11426 tablet_bootstrap.cc:492] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016: Bootstrap starting.
14:02:54.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.423686 11422 log.cc:826] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196: Log is configured to *not* fsync() on all Append() calls
14:02:54.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.424253 11426 tablet_bootstrap.cc:654] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016: Neither blocks nor log segments found. Creating new log.
14:02:54.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.424650 11426 log.cc:826] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016: Log is configured to *not* fsync() on all Append() calls
14:02:54.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.425277 11426 tablet_bootstrap.cc:492] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016: No bootstrap required, opened a new log
14:02:54.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.425333 11426 ts_tablet_manager.cc:1403] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016: Time spent bootstrapping tablet: real 0.002s	user 0.001s	sys 0.000s
14:02:54.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.427047 11426 raft_consensus.cc:359] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:02:54.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.427181 11426 raft_consensus.cc:385] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
14:02:54.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.427210 11426 raft_consensus.cc:740] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 751f3be0b90f4a1d86af88ca13d42016, State: Initialized, Role: FOLLOWER
14:02:54.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.427237 11421 tablet_bootstrap.cc:492] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962: No bootstrap required, opened a new log
14:02:54.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.427294 11421 ts_tablet_manager.cc:1403] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962: Time spent bootstrapping tablet: real 0.007s	user 0.001s	sys 0.000s
14:02:54.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.427299 11426 consensus_queue.cc:260] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:02:54.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.427511 11426 ts_tablet_manager.cc:1434] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
14:02:54.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.427870 11382 heartbeater.cc:499] Master 127.10.89.125:42399 was elected leader, sending a full tablet report...
14:02:54.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.428960 11421 raft_consensus.cc:359] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:02:54.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.429088 11421 raft_consensus.cc:385] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
14:02:54.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.429122 11421 raft_consensus.cc:740] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b415ec6a9a4671b34e29775c855962, State: Initialized, Role: FOLLOWER
14:02:54.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.429322 11421 consensus_queue.cc:260] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:02:54.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.429553 11421 ts_tablet_manager.cc:1434] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
14:02:54.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:54.433642 11383 tablet.cc:2406] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016: Can't schedule compaction. Clean time has not been advanced past its initial value.
14:02:54.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.439218 11422 tablet_bootstrap.cc:492] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196: No bootstrap required, opened a new log
14:02:54.439 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:894) Opened table 7eede73d066f4e5491d4fa0ef0f8a051
14:02:54.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.439607 11422 ts_tablet_manager.cc:1403] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196: Time spent bootstrapping tablet: real 0.018s	user 0.002s	sys 0.000s
14:02:54.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.441819 11422 raft_consensus.cc:359] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:02:54.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.442024 11422 raft_consensus.cc:385] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
14:02:54.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.442063 11422 raft_consensus.cc:740] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bf0930626894cff94471c02ef2b9196, State: Initialized, Role: FOLLOWER
14:02:54.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.442193 11422 consensus_queue.cc:260] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:02:54.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.442487 11422 ts_tablet_manager.cc:1434] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
14:02:54.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:54.551338 11230 tablet.cc:2406] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196: Can't schedule compaction. Clean time has not been advanced past its initial value.
14:02:54.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.630779 11430 raft_consensus.cc:493] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
14:02:54.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.630955 11430 raft_consensus.cc:515] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:02:54.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.631392 11430 leader_election.cc:290] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8b415ec6a9a4671b34e29775c855962 (127.10.89.65:36825), 7bf0930626894cff94471c02ef2b9196 (127.10.89.66:34685)
14:02:54.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.635103 11174 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28da754494014b9f85434e4ebbbef9d7" candidate_uuid: "751f3be0b90f4a1d86af88ca13d42016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bf0930626894cff94471c02ef2b9196" is_pre_election: true
14:02:54.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.635293 11174 raft_consensus.cc:2468] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 751f3be0b90f4a1d86af88ca13d42016 in term 0.
14:02:54.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.635538 11249 leader_election.cc:304] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 751f3be0b90f4a1d86af88ca13d42016, 7bf0930626894cff94471c02ef2b9196; no voters: 
14:02:54.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.635690 11430 raft_consensus.cc:2804] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 0 FOLLOWER]: Leader pre-election won for term 1
14:02:54.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.635735 11430 raft_consensus.cc:493] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
14:02:54.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.635756 11430 raft_consensus.cc:3060] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 0 FOLLOWER]: Advancing to term 1
14:02:54.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.635823 11018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28da754494014b9f85434e4ebbbef9d7" candidate_uuid: "751f3be0b90f4a1d86af88ca13d42016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b415ec6a9a4671b34e29775c855962" is_pre_election: true
14:02:54.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.635998 11018 raft_consensus.cc:2468] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 751f3be0b90f4a1d86af88ca13d42016 in term 0.
14:02:54.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.636623 11430 raft_consensus.cc:515] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:02:54.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.636793 11430 leader_election.cc:290] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [CANDIDATE]: Term 1 election: Requested vote from peers d8b415ec6a9a4671b34e29775c855962 (127.10.89.65:36825), 7bf0930626894cff94471c02ef2b9196 (127.10.89.66:34685)
14:02:54.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.637153 11018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28da754494014b9f85434e4ebbbef9d7" candidate_uuid: "751f3be0b90f4a1d86af88ca13d42016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b415ec6a9a4671b34e29775c855962"
14:02:54.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.637239 11018 raft_consensus.cc:3060] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 0 FOLLOWER]: Advancing to term 1
14:02:54.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.637677 11174 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28da754494014b9f85434e4ebbbef9d7" candidate_uuid: "751f3be0b90f4a1d86af88ca13d42016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bf0930626894cff94471c02ef2b9196"
14:02:54.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.637771 11174 raft_consensus.cc:3060] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 0 FOLLOWER]: Advancing to term 1
14:02:54.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.638661 11174 raft_consensus.cc:2468] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 751f3be0b90f4a1d86af88ca13d42016 in term 1.
14:02:54.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.638914 11249 leader_election.cc:304] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 751f3be0b90f4a1d86af88ca13d42016, 7bf0930626894cff94471c02ef2b9196; no voters: 
14:02:54.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.639005 11430 raft_consensus.cc:2804] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 1 FOLLOWER]: Leader election won for term 1
14:02:54.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.639086 11018 raft_consensus.cc:2468] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 751f3be0b90f4a1d86af88ca13d42016 in term 1.
14:02:54.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.639178 11430 raft_consensus.cc:697] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 1 LEADER]: Becoming Leader. State: Replica: 751f3be0b90f4a1d86af88ca13d42016, State: Running, Role: LEADER
14:02:54.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.639277 11430 consensus_queue.cc:237] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:02:54.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.640093 10732 catalog_manager.cc:5719] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 reported cstate change: term changed from 0 to 1, leader changed from <none> to 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67). New cstate: current_term: 1 leader_uuid: "751f3be0b90f4a1d86af88ca13d42016" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } health_report { overall_health: UNKNOWN } } }
14:02:54.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:54.666355 11078 tablet.cc:2406] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962: Can't schedule compaction. Clean time has not been advanced past its initial value.
14:02:54.690 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:894) Opened table 7eede73d066f4e5491d4fa0ef0f8a051
14:02:54.711 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:02:54.717 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299994, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)[L],7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685),d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)]}]
14:02:54.718 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=null, attempt=1, sequence_id=-1, TimeoutTracker(timeout=30000, elapsed=17), Traces: [0ms] refreshing cache from master, [1ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [9ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=null) after lookup
14:02:54.728 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] connecting to peer
14:02:54.731 [DEBUG - kudu-nio-3] (Connection.java:757) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] Successfully connected to peer
14:02:54.813 [DEBUG - kudu-nio-3] (Negotiator.java:918) Authenticated connection [id: 0xbea970e1, L:/127.0.0.1:60852 - R:/127.10.89.67:46723] using TOKEN/null
14:02:54.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.816640 11018 raft_consensus.cc:1275] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 1 FOLLOWER]: Refusing update from remote peer 751f3be0b90f4a1d86af88ca13d42016: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
14:02:54.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.816629 11174 raft_consensus.cc:1275] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 1 FOLLOWER]: Refusing update from remote peer 751f3be0b90f4a1d86af88ca13d42016: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
14:02:54.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.817332 11430 consensus_queue.cc:1048] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
14:02:54.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.817484 11435 consensus_queue.cc:1048] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
14:02:54.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.822739 11439 mvcc.cc:204] Tried to move back new op lower bound from 7317327768845086720 to 7317327768123068416. Current Snapshot: MvccSnapshot[applied={T|T < 7317327768845086720}]
14:02:54.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.823364 11440 mvcc.cc:204] Tried to move back new op lower bound from 7317327768845086720 to 7317327768123068416. Current Snapshot: MvccSnapshot[applied={T|T < 7317327768845086720}]
14:02:54.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.826898 11441 mvcc.cc:204] Tried to move back new op lower bound from 7317327768845086720 to 7317327768123068416. Current Snapshot: MvccSnapshot[applied={T|T < 7317327768845086720 or (T in {7317327768845086720})}]
14:02:54.913 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 3 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823, tablet=28da754494014b9f85434e4ebbbef9d7, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:02:54.915 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.10.89.67:46723
14:02:54.917 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

14:02:54.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:54.917801 10597 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskhBFXjG/build/release/bin/kudu with pid 11234
14:02:54.929 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:02:54.932 [DEBUG - kudu-nio-3] (Connection.java:239) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] handling channelInactive
14:02:54.932 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] cleaning up while in state READY due to: connection closed
14:02:54.933 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723) for tablet 28da754494014b9f85434e4ebbbef9d7: connection closed
14:02:54.948 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:02:54.948 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)[L],7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685),d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)]}]
14:02:54.948 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=2, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=18), Traces: [0ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [3ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [18ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@22118981(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@5516620b -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6e97fc17 -> wakeup thread Time-limited test)) after lookup
14:02:54.949 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] connecting to peer
14:02:54.952 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:02:54.953 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] cleaning up while in state CONNECTING due to: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:02:54.953 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:02:54.967 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:02:54.967 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)[L],7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685),d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)]}]
14:02:54.968 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=4, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=37), Traces: [0ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [3ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [18ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [20ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [24ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [24ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [37ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@22118981(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@5516620b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1964384961 after retry RPC@1287315673), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6e97fc17 -> wakeup thread Time-limited test -> (continuation of Deferred@1964384961 after retry RPC@1287315673))) after lookup
14:02:54.968 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] connecting to peer
14:02:54.970 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:02:54.971 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] cleaning up while in state CONNECTING due to: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:02:54.971 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:02:55.008 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:02:55.009 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)[L],7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685),d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)]}]
14:02:55.010 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=6, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=79), Traces: [0ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [3ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [18ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [20ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [24ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [24ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [37ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [39ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [42ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [42ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [76ms] refreshing cache from master, [76ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [79ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@22118981(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@5516620b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1964384961 after retry RPC@1287315673) -> (continuation of Deferred@1954378175 after retry RPC@1822211443), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6e97fc17 -> wakeup thread Time-limited test -> (continuation of Deferred@1964384961 after retry RPC@1287315673) -> (continuation of Deferred@1954378175 after retry RPC@1822211443))) after lookup
14:02:55.010 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] connecting to peer
14:02:55.012 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:02:55.013 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] cleaning up while in state CONNECTING due to: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:02:55.013 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:02:55.067 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:02:55.068 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)[L],7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685),d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)]}]
14:02:55.068 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=8, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=138), Traces: [0ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [3ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [18ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [20ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [24ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [24ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [37ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [39ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [42ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [42ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [76ms] refreshing cache from master, [76ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [79ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [81ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [84ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [84ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [135ms] refreshing cache from master, [135ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [138ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@22118981(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@5516620b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1964384961 after retry RPC@1287315673) -> (continuation of Deferred@1954378175 after retry RPC@1822211443) -> (continuation of Deferred@533776822 after retry RPC@535937942), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6e97fc17 -> wakeup thread Time-limited test -> (continuation of Deferred@1964384961 after retry RPC@1287315673) -> (continuation of Deferred@1954378175 after retry RPC@1822211443) -> (continuation of Deferred@533776822 after retry RPC@535937942))) after lookup
14:02:55.069 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] connecting to peer
14:02:55.070 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:02:55.071 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] cleaning up while in state CONNECTING due to: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:02:55.071 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:02:55.267 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:02:55.268 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)[L],7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685),d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)]}]
14:02:55.269 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=10, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=338), Traces: [0ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [3ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [18ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [20ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [24ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [24ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [37ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [39ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [42ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [42ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [76ms] refreshing cache from master, [76ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [79ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [81ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [84ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [84ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [135ms] refreshing cache from master, [135ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [138ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [139ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [142ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [142ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [335ms] refreshing cache from master, [335ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [338ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@22118981(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@5516620b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1964384961 after retry RPC@1287315673) -> (continuation of Deferred@1954378175 after retry RPC@1822211443) -> (continuation of Deferred@533776822 after retry RPC@535937942) -> (continuation of Deferred@1933712835 after retry RPC@328648240), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6e97fc17 -> wakeup thread Time-limited test -> (continuation of Deferred@1964384961 after retry RPC@1287315673) -> (continuation of Deferred@1954378175 after retry RPC@1822211443) -> (continuation of Deferred@533776822 after retry RPC@535937942) -> (continuation of Deferred@1933712835 after retry RPC@328648240))) after lookup
14:02:55.270 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] connecting to peer
14:02:55.271 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:02:55.271 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] cleaning up while in state CONNECTING due to: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:02:55.271 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:02:56.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:56.356925 11455 raft_consensus.cc:493] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 751f3be0b90f4a1d86af88ca13d42016)
14:02:56.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:56.357033 11455 raft_consensus.cc:515] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:02:56.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:56.357532 11457 raft_consensus.cc:493] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 751f3be0b90f4a1d86af88ca13d42016)
14:02:56.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:56.357641 11457 raft_consensus.cc:515] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:02:56.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:56.357782 11455 leader_election.cc:290] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723), 7bf0930626894cff94471c02ef2b9196 (127.10.89.66:34685)
14:02:56.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:56.357949 11457 leader_election.cc:290] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723), d8b415ec6a9a4671b34e29775c855962 (127.10.89.65:36825)
14:02:56.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:56.358397 10943 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.89.67:46723: connect: Connection refused (error 111)
14:02:56.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:56.358526 11100 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.89.67:46723: connect: Connection refused (error 111)
14:02:56.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:56.359032 11100 leader_election.cc:336] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723): Network error: Client connection negotiation failed: client connection to 127.10.89.67:46723: connect: Connection refused (error 111)
14:02:56.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:56.359411 10943 leader_election.cc:336] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723): Network error: Client connection negotiation failed: client connection to 127.10.89.67:46723: connect: Connection refused (error 111)
14:02:56.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:56.361331 11174 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28da754494014b9f85434e4ebbbef9d7" candidate_uuid: "d8b415ec6a9a4671b34e29775c855962" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7bf0930626894cff94471c02ef2b9196" is_pre_election: true
14:02:56.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:56.361438 11174 raft_consensus.cc:2468] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8b415ec6a9a4671b34e29775c855962 in term 1.
14:02:56.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:56.361681 10940 leader_election.cc:304] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7bf0930626894cff94471c02ef2b9196, d8b415ec6a9a4671b34e29775c855962; no voters: 751f3be0b90f4a1d86af88ca13d42016
14:02:56.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:56.361758 11018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28da754494014b9f85434e4ebbbef9d7" candidate_uuid: "7bf0930626894cff94471c02ef2b9196" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8b415ec6a9a4671b34e29775c855962" is_pre_election: true
14:02:56.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:56.361831 11455 raft_consensus.cc:2804] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 1 FOLLOWER]: Leader pre-election won for term 2
14:02:56.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:56.361871 11018 raft_consensus.cc:2468] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bf0930626894cff94471c02ef2b9196 in term 1.
14:02:56.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:56.361892 11455 raft_consensus.cc:493] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 751f3be0b90f4a1d86af88ca13d42016)
14:02:56.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:56.361914 11455 raft_consensus.cc:3060] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 1 FOLLOWER]: Advancing to term 2
14:02:56.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:56.362066 11100 leader_election.cc:304] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7bf0930626894cff94471c02ef2b9196, d8b415ec6a9a4671b34e29775c855962; no voters: 751f3be0b90f4a1d86af88ca13d42016
14:02:56.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:56.362197 11457 raft_consensus.cc:2804] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 1 FOLLOWER]: Leader pre-election won for term 2
14:02:56.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:56.362251 11457 raft_consensus.cc:493] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 751f3be0b90f4a1d86af88ca13d42016)
14:02:56.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:56.362273 11457 raft_consensus.cc:3060] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 1 FOLLOWER]: Advancing to term 2
14:02:56.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:56.362753 11455 raft_consensus.cc:515] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:02:56.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:56.362914 11455 leader_election.cc:290] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [CANDIDATE]: Term 2 election: Requested vote from peers 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723), 7bf0930626894cff94471c02ef2b9196 (127.10.89.66:34685)
14:02:56.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:56.363273 11174 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28da754494014b9f85434e4ebbbef9d7" candidate_uuid: "d8b415ec6a9a4671b34e29775c855962" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7bf0930626894cff94471c02ef2b9196"
14:02:56.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:56.363241 11457 raft_consensus.cc:515] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:02:56.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:56.363503 10943 leader_election.cc:336] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723): Network error: Client connection negotiation failed: client connection to 127.10.89.67:46723: connect: Connection refused (error 111)
14:02:56.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:56.363653 11174 raft_consensus.cc:2393] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d8b415ec6a9a4671b34e29775c855962 in current term 2: Already voted for candidate 7bf0930626894cff94471c02ef2b9196 in this term.
14:02:56.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:56.363852 10940 leader_election.cc:304] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8b415ec6a9a4671b34e29775c855962; no voters: 751f3be0b90f4a1d86af88ca13d42016, 7bf0930626894cff94471c02ef2b9196
14:02:56.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:56.363967 11455 raft_consensus.cc:2749] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
14:02:56.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:56.364054 11100 leader_election.cc:336] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723): Network error: Client connection negotiation failed: client connection to 127.10.89.67:46723: connect: Connection refused (error 111)
14:02:56.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:56.363981 11457 leader_election.cc:290] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [CANDIDATE]: Term 2 election: Requested vote from peers 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723), d8b415ec6a9a4671b34e29775c855962 (127.10.89.65:36825)
14:02:56.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:56.364174 11018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28da754494014b9f85434e4ebbbef9d7" candidate_uuid: "7bf0930626894cff94471c02ef2b9196" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8b415ec6a9a4671b34e29775c855962"
14:02:56.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:56.364248 11018 raft_consensus.cc:2393] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7bf0930626894cff94471c02ef2b9196 in current term 2: Already voted for candidate d8b415ec6a9a4671b34e29775c855962 in this term.
14:02:56.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:56.364434 11100 leader_election.cc:304] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7bf0930626894cff94471c02ef2b9196; no voters: 751f3be0b90f4a1d86af88ca13d42016, d8b415ec6a9a4671b34e29775c855962
14:02:56.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:56.364531 11457 raft_consensus.cc:2749] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
14:02:57.247 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:02:57.248 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)[L],7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685),d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)]}]
14:02:57.249 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=12, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=2318), Traces: [0ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [3ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [18ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [20ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [24ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [24ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [37ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [39ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [42ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [42ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [76ms] refreshing cache from master, [76ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [79ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [81ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [84ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [84ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [135ms] refreshing cache from master, [135ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [138ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [139ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [142ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [142ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [335ms] refreshing cache from master, [335ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [338ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [340ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [342ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [342ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [2315ms] refreshing cache from master, [2316ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [2318ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@22118981(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@5516620b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1964384961 after retry RPC@1287315673) -> (continuation of Deferred@1954378175 after retry RPC@1822211443) -> (continuation of Deferred@533776822 after retry RPC@535937942) -> (continuation of Deferred@1933712835 after retry RPC@328648240) -> (continuation of Deferred@1321082189 after retry RPC@1799153132), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6e97fc17 -> wakeup thread Time-limited test -> (continuation of Deferred@1964384961 after retry RPC@1287315673) -> (continuation of Deferred@1954378175 after retry RPC@1822211443) -> (continuation of Deferred@533776822 after retry RPC@535937942) -> (continuation of Deferred@1933712835 after retry RPC@328648240) -> (continuation of Deferred@1321082189 after retry RPC@1799153132))) after lookup
14:02:57.250 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] connecting to peer
14:02:57.251 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:02:57.251 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] cleaning up while in state CONNECTING due to: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:02:57.251 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:02:57.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:57.915738 11469 raft_consensus.cc:493] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
14:02:57.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:57.915848 11469 raft_consensus.cc:515] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:02:57.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:57.916057 11469 leader_election.cc:290] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723), d8b415ec6a9a4671b34e29775c855962 (127.10.89.65:36825)
14:02:57.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:57.916558 11018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28da754494014b9f85434e4ebbbef9d7" candidate_uuid: "7bf0930626894cff94471c02ef2b9196" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8b415ec6a9a4671b34e29775c855962" is_pre_election: true
14:02:57.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:57.916663 11018 raft_consensus.cc:2468] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bf0930626894cff94471c02ef2b9196 in term 2.
14:02:57.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:57.916841 11100 leader_election.cc:304] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bf0930626894cff94471c02ef2b9196, d8b415ec6a9a4671b34e29775c855962; no voters: 
14:02:57.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:57.916936 11100 leader_election.cc:336] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723): Network error: Client connection negotiation failed: client connection to 127.10.89.67:46723: connect: Connection refused (error 111)
14:02:57.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:57.916944 11469 raft_consensus.cc:2804] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 2 FOLLOWER]: Leader pre-election won for term 3
14:02:57.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:57.916998 11469 raft_consensus.cc:493] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
14:02:57.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:57.917021 11469 raft_consensus.cc:3060] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 2 FOLLOWER]: Advancing to term 3
14:02:57.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:57.917655 11469 raft_consensus.cc:515] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:02:57.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:57.917791 11469 leader_election.cc:290] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [CANDIDATE]: Term 3 election: Requested vote from peers 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723), d8b415ec6a9a4671b34e29775c855962 (127.10.89.65:36825)
14:02:57.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:57.918067 11018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28da754494014b9f85434e4ebbbef9d7" candidate_uuid: "7bf0930626894cff94471c02ef2b9196" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8b415ec6a9a4671b34e29775c855962"
14:02:57.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:57.918155 11018 raft_consensus.cc:3060] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 2 FOLLOWER]: Advancing to term 3
14:02:57.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:57.918531 11100 leader_election.cc:336] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723): Network error: Client connection negotiation failed: client connection to 127.10.89.67:46723: connect: Connection refused (error 111)
14:02:57.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:57.918951 11018 raft_consensus.cc:2468] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bf0930626894cff94471c02ef2b9196 in term 3.
14:02:57.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:57.919176 11100 leader_election.cc:304] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7bf0930626894cff94471c02ef2b9196, d8b415ec6a9a4671b34e29775c855962; no voters: 751f3be0b90f4a1d86af88ca13d42016
14:02:57.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:57.919266 11469 raft_consensus.cc:2804] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 3 FOLLOWER]: Leader election won for term 3
14:02:57.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:57.919374 11469 raft_consensus.cc:697] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 3 LEADER]: Becoming Leader. State: Replica: 7bf0930626894cff94471c02ef2b9196, State: Running, Role: LEADER
14:02:57.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:57.919452 11469 consensus_queue.cc:237] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:02:57.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:57.920243 10731 catalog_manager.cc:5719] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 reported cstate change: term changed from 1 to 3, leader changed from 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67) to 7bf0930626894cff94471c02ef2b9196 (127.10.89.66). New cstate: current_term: 3 leader_uuid: "7bf0930626894cff94471c02ef2b9196" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } health_report { overall_health: HEALTHY } } }
14:02:58.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:58.476264 11018 raft_consensus.cc:1275] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 3 FOLLOWER]: Refusing update from remote peer 7bf0930626894cff94471c02ef2b9196: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
14:02:58.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:02:58.476652 11486 consensus_queue.cc:1048] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
14:02:58.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:02:58.479316 11100 consensus_peers.cc:597] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 -> Peer 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723): Couldn't send request to peer 751f3be0b90f4a1d86af88ca13d42016. Status: Network error: Client connection negotiation failed: client connection to 127.10.89.67:46723: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:03:00.748 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:00.749 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723),7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)[L],d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)]}]
14:03:00.750 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=14, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=5820), Traces: [0ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [3ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: connection closed, [4ms] delaying RPC due to: Network error: connection closed, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [18ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [20ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [24ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [24ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [37ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [39ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [42ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [42ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [76ms] refreshing cache from master, [76ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [79ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [81ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [84ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [84ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [135ms] refreshing cache from master, [135ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [138ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [139ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [142ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [142ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [335ms] refreshing cache from master, [335ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [338ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [340ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [342ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [342ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [2315ms] refreshing cache from master, [2316ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [2318ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [2320ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [2322ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [2322ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [5815ms] refreshing cache from master, [5816ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [5819ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@22118981(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@5516620b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1964384961 after retry RPC@1287315673) -> (continuation of Deferred@1954378175 after retry RPC@1822211443) -> (continuation of Deferred@533776822 after retry RPC@535937942) -> (continuation of Deferred@1933712835 after retry RPC@328648240) -> (continuation of Deferred@1321082189 after retry RPC@1799153132) -> (continuation of Deferred@728526674 after retry RPC@919338543), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6e97fc17 -> wakeup thread Time-limited test -> (continuation of Deferred@1964384961 after retry RPC@1287315673) -> (continuation of Deferred@1954378175 after retry RPC@1822211443) -> (continuation of Deferred@533776822 after retry RPC@535937942) -> (continuation of Deferred@1933712835 after retry RPC@328648240) -> (continuation of Deferred@1321082189 after retry RPC@1799153132) -> (continuation of Deferred@728526674 after retry RPC@919338543))) after lookup
14:03:00.751 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] connecting to peer
14:03:00.752 [DEBUG - kudu-nio-2] (Connection.java:757) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] Successfully connected to peer
14:03:00.777 [DEBUG - kudu-nio-2] (Negotiator.java:918) Authenticated connection [id: 0x72556c9d, L:/127.0.0.1:50746 - R:/127.10.89.66:34685] using TOKEN/null
14:03:00.788 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.10.89.67:46723
14:03:00.790 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

14:03:00.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.790897 10597 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskhBFXjG/build/release/bin/kudu
14:03:00.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskhBFXjG/build/release/bin/kudu
14:03:00.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/wal
14:03:00.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/data
14:03:00.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:03:00.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:03:00.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:03:00.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:03:00.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:03:00.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:03:00.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/logs
14:03:00.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/data/info.pb
14:03:00.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:03:00.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:03:00.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:03:00.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:03:00.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:03:00.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:03:00.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:03:00.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:03:00.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:03:00.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:03:00.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.89.67:46723
14:03:00.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.89.67
14:03:00.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.89.126:36189,127.10.89.125:42399,127.10.89.124:42407
14:03:00.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34339
14:03:00.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.89.67
14:03:00.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.89.84:33487
14:03:00.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:03:00.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:03:00.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:03:00.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:00.839728 11100 consensus_peers.cc:597] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 -> Peer 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723): Couldn't send request to peer 751f3be0b90f4a1d86af88ca13d42016. Status: Network error: Client connection negotiation failed: client connection to 127.10.89.67:46723: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
14:03:00.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:00.877820 11516 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:03:00.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:00.878036 11516 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:03:00.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:00.878072 11516 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:03:00.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:00.879966 11516 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:03:00.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:00.880072 11516 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.89.67
14:03:00.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.881896 11516 tablet_server_runner.cc:78] Tablet server non-default flags:
14:03:00.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/logs
14:03:00.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:03:00.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:03:00.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:03:00.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.89.84:33487
14:03:00.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:03:00.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:03:00.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/data
14:03:00.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/wal
14:03:00.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:03:00.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:03:00.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.89.67:46723
14:03:00.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:03:00.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:03:00.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:03:00.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/data/info.pb
14:03:00.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.89.67
14:03:00.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34339
14:03:00.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.89.126:36189,127.10.89.125:42399,127.10.89.124:42407
14:03:00.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:03:00.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.11516
14:03:00.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:03:00.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:03:00.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:03:00.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:03:00.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.89.67
14:03:00.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:03:00.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:03:00.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:03:00.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 4bcd631c449d3c6649730ca6e07660c8168f0d44
14:03:00.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
14:03:00.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Aug 2026 13:43:13 UTC on e7f111948823
14:03:00.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 13289
14:03:00.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.882205 11516 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:03:00.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.882505 11516 file_cache.cc:504] Constructed file cache file cache with capacity 419430
14:03:00.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:00.886101 11524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:03:00.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:00.886097 11521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:03:00.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:00.886097 11522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:03:00.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.886260 11516 server_base.cc:1061] running on GCE node
14:03:00.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.886736 11516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:03:00.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.886938 11516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:03:00.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.888075 11516 hybrid_clock.cc:648] HybridClock initialized: now 1786456980888049 us; error 46 us; skew 500 ppm
14:03:00.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.889227 11516 webserver.cc:533] Webserver started at http://127.10.89.67:34339/ using document root <none> and password file <none>
14:03:00.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.889425 11516 fs_manager.cc:362] Metadata directory not provided
14:03:00.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.889480 11516 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:03:00.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.890759 11516 fs_manager.cc:714] Time spent opening directory manager: real 0.001s	user 0.001s	sys 0.000s
14:03:00.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.891568 11532 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:03:00.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.891788 11516 fs_manager.cc:730] Time spent opening block manager: real 0.001s	user 0.001s	sys 0.000s
14:03:00.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.891860 11516 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/data,/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/wal
14:03:00.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "751f3be0b90f4a1d86af88ca13d42016"
14:03:00.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 14:02:53 on dist-test-slave-g350"
14:03:00.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.892131 11516 fs_report.cc:389] FS layout report
14:03:00.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:03:00.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/wal
14:03:00.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/wal
14:03:00.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/data/data
14:03:00.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:03:00.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:03:00.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:03:00.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:03:00.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:03:00.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:03:00.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:03:00.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:03:00.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:03:00.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:03:00.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:03:00.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.919412 11516 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:03:00.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.919790 11516 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:03:00.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.919890 11516 kserver.cc:163] Server-wide thread pool size limit: 3276
14:03:00.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.920099 11516 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:03:00.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.920550 11539 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
14:03:00.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.921574 11516 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:03:00.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.921648 11516 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s	user 0.000s	sys 0.000s
14:03:00.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.921720 11516 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:03:00.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.922405 11516 ts_tablet_manager.cc:616] Registered 1 tablets
14:03:00.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.922439 11516 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s	user 0.000s	sys 0.000s
14:03:00.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.922545 11539 tablet_bootstrap.cc:492] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016: Bootstrap starting.
14:03:00.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.930882 11516 rpc_server.cc:307] RPC server started. Bound to: 127.10.89.67:46723
14:03:00.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.930974 11665 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.89.67:46723 every 8 connection(s)
14:03:00.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.931357 11516 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/data/info.pb
14:03:00.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.933073 11539 log.cc:826] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016: Log is configured to *not* fsync() on all Append() calls
14:03:00.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.936429 11539 tablet_bootstrap.cc:492] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:03:00.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.936753 11539 tablet_bootstrap.cc:492] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016: Bootstrap complete.
14:03:00.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.938225 11539 ts_tablet_manager.cc:1403] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016: Time spent bootstrapping tablet: real 0.016s	user 0.000s	sys 0.014s
14:03:00.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.938259 11666 heartbeater.cc:344] Connected to a master server at 127.10.89.124:42407
14:03:00.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.938346 11666 heartbeater.cc:461] Registering TS with master...
14:03:00.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.938524 11666 heartbeater.cc:507] Master 127.10.89.124:42407 requested a full tablet report, sending...
14:03:00.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.938949 11668 heartbeater.cc:344] Connected to a master server at 127.10.89.125:42399
14:03:00.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.939239 11668 heartbeater.cc:461] Registering TS with master...
14:03:00.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.939354 10814 ts_manager.cc:194] Re-registered known tserver with Master: 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723)
14:03:00.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.939553 11668 heartbeater.cc:507] Master 127.10.89.125:42399 requested a full tablet report, sending...
14:03:00.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.939438 11539 raft_consensus.cc:359] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:00.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.939703 11539 raft_consensus.cc:740] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 751f3be0b90f4a1d86af88ca13d42016, State: Initialized, Role: FOLLOWER
14:03:00.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.940078 10597 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskhBFXjG/build/release/bin/kudu as pid 11516
14:03:00.940 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:03:00.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.941159 11539 consensus_queue.cc:260] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:00.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.941490 11539 ts_tablet_manager.cc:1434] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
14:03:00.945 [DEBUG - kudu-nio-2] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 6 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823, tablet=28da754494014b9f85434e4ebbbef9d7, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:03:00.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.944602 10731 ts_manager.cc:194] Re-registered known tserver with Master: 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723)
14:03:00.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.945731 10731 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.10.89.67:33141
14:03:00.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.945937 11667 heartbeater.cc:344] Connected to a master server at 127.10.89.126:36189
14:03:00.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.946000 11667 heartbeater.cc:461] Registering TS with master...
14:03:00.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.946096 11667 heartbeater.cc:507] Master 127.10.89.126:36189 requested a full tablet report, sending...
14:03:00.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.946485 11668 heartbeater.cc:499] Master 127.10.89.125:42399 was elected leader, sending a full tablet report...
14:03:00.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.946661 10644 ts_manager.cc:194] Re-registered known tserver with Master: 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723)
14:03:00.947 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.10.89.66:34685
14:03:00.948 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

14:03:00.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:00.948583 10597 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskhBFXjG/build/release/bin/kudu with pid 11082
14:03:00.962 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:03:00.962 [DEBUG - kudu-nio-2] (Connection.java:239) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] handling channelInactive
14:03:00.962 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] cleaning up while in state READY due to: connection closed
14:03:00.985 [DEBUG - pool-2-thread-1] (Connection.java:748) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] connecting to peer
14:03:00.986 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:00.986 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:00.987 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:01.007 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:01.007 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723),7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)[L],d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)]}]
14:03:01.007 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=3, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=45), Traces: [0ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [0ms] delaying RPC due to: Illegal state: connection is terminated, [22ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [25ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [25ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [42ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [44ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@244614442(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@67c73851 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@39d2a6f2 -> wakeup thread Time-limited test)) after lookup
14:03:01.008 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] connecting to peer
14:03:01.009 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:01.010 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:01.010 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:01.027 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:01.028 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723),7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)[L],d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)]}]
14:03:01.028 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=5, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=66), Traces: [0ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [0ms] delaying RPC due to: Illegal state: connection is terminated, [22ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [25ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [25ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [42ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [44ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [46ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [48ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [48ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [63ms] refreshing cache from master, [63ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [65ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@244614442(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@67c73851 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1491310046 after retry RPC@1948933382), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@39d2a6f2 -> wakeup thread Time-limited test -> (continuation of Deferred@1491310046 after retry RPC@1948933382))) after lookup
14:03:01.029 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] connecting to peer
14:03:01.030 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:01.030 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:01.030 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:01.088 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:01.089 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723),7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)[L],d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)]}]
14:03:01.090 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=7, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=127), Traces: [0ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [0ms] delaying RPC due to: Illegal state: connection is terminated, [22ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [25ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [25ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [42ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [44ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [46ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [48ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [48ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [63ms] refreshing cache from master, [63ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [65ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [67ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [68ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [68ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [122ms] refreshing cache from master, [123ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [125ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@244614442(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@67c73851 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1491310046 after retry RPC@1948933382) -> (continuation of Deferred@567157376 after retry RPC@128616914), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@39d2a6f2 -> wakeup thread Time-limited test -> (continuation of Deferred@1491310046 after retry RPC@1948933382) -> (continuation of Deferred@567157376 after retry RPC@128616914))) after lookup
14:03:01.091 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] connecting to peer
14:03:01.093 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:01.093 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:01.093 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:01.267 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:01.268 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723),7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)[L],d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)]}]
14:03:01.268 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=9, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=306), Traces: [0ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [0ms] delaying RPC due to: Illegal state: connection is terminated, [22ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [25ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [25ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [42ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [44ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [46ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [48ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [48ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [63ms] refreshing cache from master, [63ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [65ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [67ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [68ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [68ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [122ms] refreshing cache from master, [123ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [125ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [128ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [131ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [131ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [302ms] refreshing cache from master, [303ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [305ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@244614442(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@67c73851 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1491310046 after retry RPC@1948933382) -> (continuation of Deferred@567157376 after retry RPC@128616914) -> (continuation of Deferred@1375960924 after retry RPC@20220119), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@39d2a6f2 -> wakeup thread Time-limited test -> (continuation of Deferred@1491310046 after retry RPC@1948933382) -> (continuation of Deferred@567157376 after retry RPC@128616914) -> (continuation of Deferred@1375960924 after retry RPC@20220119))) after lookup
14:03:01.269 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] connecting to peer
14:03:01.271 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:01.271 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:01.271 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:01.487 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:01.488 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723),7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)[L],d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)]}]
14:03:01.490 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=11, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=527), Traces: [0ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [0ms] delaying RPC due to: Illegal state: connection is terminated, [22ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [25ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [25ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [42ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [44ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [46ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [48ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [48ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [63ms] refreshing cache from master, [63ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [65ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [67ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [68ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [68ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [122ms] refreshing cache from master, [123ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [125ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [128ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [131ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [131ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [302ms] refreshing cache from master, [303ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [305ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [307ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [309ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [309ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [523ms] refreshing cache from master, [523ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [525ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@244614442(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@67c73851 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1491310046 after retry RPC@1948933382) -> (continuation of Deferred@567157376 after retry RPC@128616914) -> (continuation of Deferred@1375960924 after retry RPC@20220119) -> (continuation of Deferred@1838155989 after retry RPC@1597629717), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@39d2a6f2 -> wakeup thread Time-limited test -> (continuation of Deferred@1491310046 after retry RPC@1948933382) -> (continuation of Deferred@567157376 after retry RPC@128616914) -> (continuation of Deferred@1375960924 after retry RPC@20220119) -> (continuation of Deferred@1838155989 after retry RPC@1597629717))) after lookup
14:03:01.490 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] connecting to peer
14:03:01.492 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:01.492 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:01.492 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:02.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:02.288089 11682 raft_consensus.cc:493] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7bf0930626894cff94471c02ef2b9196)
14:03:02.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:02.288189 11682 raft_consensus.cc:515] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:02.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:02.288466 11682 leader_election.cc:290] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723), 7bf0930626894cff94471c02ef2b9196 (127.10.89.66:34685)
14:03:02.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:02.289359 10940 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.89.66:34685: connect: Connection refused (error 111) [suppressed 1 similar messages]
14:03:02.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:02.289943 10940 leader_election.cc:336] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 7bf0930626894cff94471c02ef2b9196 (127.10.89.66:34685): Network error: Client connection negotiation failed: client connection to 127.10.89.66:34685: connect: Connection refused (error 111)
14:03:02.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:02.291994 11615 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28da754494014b9f85434e4ebbbef9d7" candidate_uuid: "d8b415ec6a9a4671b34e29775c855962" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: false dest_uuid: "751f3be0b90f4a1d86af88ca13d42016" is_pre_election: true
14:03:02.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:02.292168 11615 raft_consensus.cc:2468] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8b415ec6a9a4671b34e29775c855962 in term 1.
14:03:02.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:02.292526 10943 leader_election.cc:304] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 751f3be0b90f4a1d86af88ca13d42016, d8b415ec6a9a4671b34e29775c855962; no voters: 7bf0930626894cff94471c02ef2b9196
14:03:02.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:02.292768 11682 raft_consensus.cc:2804] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 3 FOLLOWER]: Leader pre-election won for term 4
14:03:02.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:02.292824 11682 raft_consensus.cc:493] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7bf0930626894cff94471c02ef2b9196)
14:03:02.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:02.292845 11682 raft_consensus.cc:3060] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 3 FOLLOWER]: Advancing to term 4
14:03:02.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:02.293568 11682 raft_consensus.cc:515] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:02.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:02.293799 11682 leader_election.cc:290] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [CANDIDATE]: Term 4 election: Requested vote from peers 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723), 7bf0930626894cff94471c02ef2b9196 (127.10.89.66:34685)
14:03:02.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:02.294049 11615 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28da754494014b9f85434e4ebbbef9d7" candidate_uuid: "d8b415ec6a9a4671b34e29775c855962" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: false dest_uuid: "751f3be0b90f4a1d86af88ca13d42016"
14:03:02.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:02.294188 11615 raft_consensus.cc:3060] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 1 FOLLOWER]: Advancing to term 4
14:03:02.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:02.294476 10940 leader_election.cc:336] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 7bf0930626894cff94471c02ef2b9196 (127.10.89.66:34685): Network error: Client connection negotiation failed: client connection to 127.10.89.66:34685: connect: Connection refused (error 111)
14:03:02.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:02.295282 11615 raft_consensus.cc:2468] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8b415ec6a9a4671b34e29775c855962 in term 4.
14:03:02.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:02.295480 10943 leader_election.cc:304] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 751f3be0b90f4a1d86af88ca13d42016, d8b415ec6a9a4671b34e29775c855962; no voters: 7bf0930626894cff94471c02ef2b9196
14:03:02.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:02.295588 11682 raft_consensus.cc:2804] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 4 FOLLOWER]: Leader election won for term 4
14:03:02.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:02.295758 11682 raft_consensus.cc:697] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 4 LEADER]: Becoming Leader. State: Replica: d8b415ec6a9a4671b34e29775c855962, State: Running, Role: LEADER
14:03:02.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:02.295905 11682 consensus_queue.cc:237] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.8, Last appended by leader: 8, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:02.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:02.296707 10731 catalog_manager.cc:5719] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 reported cstate change: term changed from 3 to 4, leader changed from 7bf0930626894cff94471c02ef2b9196 (127.10.89.66) to d8b415ec6a9a4671b34e29775c855962 (127.10.89.65). New cstate: current_term: 4 leader_uuid: "d8b415ec6a9a4671b34e29775c855962" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } health_report { overall_health: UNKNOWN } } }
14:03:02.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:02.792900 10940 consensus_peers.cc:597] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 -> Peer 7bf0930626894cff94471c02ef2b9196 (127.10.89.66:34685): Couldn't send request to peer 7bf0930626894cff94471c02ef2b9196. Status: Network error: Client connection negotiation failed: client connection to 127.10.89.66:34685: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:03:02.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:02.802513 11615 raft_consensus.cc:1275] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 4 FOLLOWER]: Refusing update from remote peer d8b415ec6a9a4671b34e29775c855962: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 9. (index mismatch)
14:03:02.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:02.802870 11682 consensus_queue.cc:1048] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [LEADER]: Connected to new peer: Peer: permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 4, Time since last communication: 0.000s
14:03:03.548 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:03.548 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723),7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685),d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)[L]]}]
14:03:03.549 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=13, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=2586), Traces: [0ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [0ms] delaying RPC due to: Illegal state: connection is terminated, [22ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [25ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [25ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [42ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [44ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [46ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [48ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [48ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [63ms] refreshing cache from master, [63ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [65ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [67ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [68ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [68ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [122ms] refreshing cache from master, [123ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [125ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [128ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [131ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [131ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [302ms] refreshing cache from master, [303ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [305ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [307ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [309ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [309ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [523ms] refreshing cache from master, [523ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [525ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [528ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [530ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [530ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [2583ms] refreshing cache from master, [2583ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [2585ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@244614442(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@67c73851 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1491310046 after retry RPC@1948933382) -> (continuation of Deferred@567157376 after retry RPC@128616914) -> (continuation of Deferred@1375960924 after retry RPC@20220119) -> (continuation of Deferred@1838155989 after retry RPC@1597629717) -> (continuation of Deferred@758899726 after retry RPC@1027259889), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@39d2a6f2 -> wakeup thread Time-limited test -> (continuation of Deferred@1491310046 after retry RPC@1948933382) -> (continuation of Deferred@567157376 after retry RPC@128616914) -> (continuation of Deferred@1375960924 after retry RPC@20220119) -> (continuation of Deferred@1838155989 after retry RPC@1597629717) -> (continuation of Deferred@758899726 after retry RPC@1027259889))) after lookup
14:03:03.550 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] connecting to peer
14:03:03.551 [DEBUG - kudu-nio-1] (Connection.java:757) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] Successfully connected to peer
14:03:03.574 [DEBUG - kudu-nio-1] (Negotiator.java:918) Authenticated connection [id: 0x3232f32b, L:/127.0.0.1:53516 - R:/127.10.89.65:36825] using TOKEN/null
14:03:03.585 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.10.89.66:34685
14:03:03.586 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

14:03:03.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.586944 10597 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskhBFXjG/build/release/bin/kudu
14:03:03.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskhBFXjG/build/release/bin/kudu
14:03:03.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/wal
14:03:03.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/data
14:03:03.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:03:03.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:03:03.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:03:03.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:03:03.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:03:03.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:03:03.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/logs
14:03:03.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/data/info.pb
14:03:03.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:03:03.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:03:03.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:03:03.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:03:03.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:03:03.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:03:03.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:03:03.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:03:03.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:03:03.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:03:03.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.89.66:34685
14:03:03.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.89.66
14:03:03.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.89.126:36189,127.10.89.125:42399,127.10.89.124:42407
14:03:03.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=39225
14:03:03.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.89.66
14:03:03.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.89.84:33487
14:03:03.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:03:03.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:03:03.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:03:03.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:03.672027 11718 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:03:03.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:03.672258 11718 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:03:03.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:03.672307 11718 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:03:03.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:03.674201 11718 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:03:03.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:03.674265 11718 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.89.66
14:03:03.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.675850 11718 tablet_server_runner.cc:78] Tablet server non-default flags:
14:03:03.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/logs
14:03:03.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:03:03.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:03:03.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:03:03.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.89.84:33487
14:03:03.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:03:03.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:03:03.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/data
14:03:03.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/wal
14:03:03.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:03:03.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:03:03.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.89.66:34685
14:03:03.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:03:03.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:03:03.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:03:03.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/data/info.pb
14:03:03.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.89.66
14:03:03.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=39225
14:03:03.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.89.126:36189,127.10.89.125:42399,127.10.89.124:42407
14:03:03.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:03:03.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.11718
14:03:03.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:03:03.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:03:03.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:03:03.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:03:03.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.89.66
14:03:03.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:03:03.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:03:03.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:03:03.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 4bcd631c449d3c6649730ca6e07660c8168f0d44
14:03:03.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
14:03:03.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Aug 2026 13:43:13 UTC on e7f111948823
14:03:03.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 13289
14:03:03.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.676079 11718 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:03:03.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.676275 11718 file_cache.cc:504] Constructed file cache file cache with capacity 419430
14:03:03.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:03.679136 11726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:03:03.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:03.679103 11725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:03:03.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:03.679126 11728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:03:03.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.679631 11718 server_base.cc:1061] running on GCE node
14:03:03.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.679898 11718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:03:03.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.680161 11718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:03:03.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.681336 11718 hybrid_clock.cc:648] HybridClock initialized: now 1786456983681288 us; error 58 us; skew 500 ppm
14:03:03.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.682652 11718 webserver.cc:533] Webserver started at http://127.10.89.66:39225/ using document root <none> and password file <none>
14:03:03.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.682889 11718 fs_manager.cc:362] Metadata directory not provided
14:03:03.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.682968 11718 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:03:03.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.684209 11718 fs_manager.cc:714] Time spent opening directory manager: real 0.001s	user 0.001s	sys 0.000s
14:03:03.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.684930 11735 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:03:03.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.685119 11718 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.001s	sys 0.000s
14:03:03.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.685202 11718 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/data,/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/wal
14:03:03.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "7bf0930626894cff94471c02ef2b9196"
14:03:03.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 14:02:53 on dist-test-slave-g350"
14:03:03.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.685539 11718 fs_report.cc:389] FS layout report
14:03:03.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:03:03.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/wal
14:03:03.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/wal
14:03:03.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/data/data
14:03:03.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:03:03.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:03:03.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:03:03.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:03:03.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:03:03.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:03:03.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:03:03.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:03:03.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:03:03.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:03:03.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:03:03.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.696667 11718 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:03:03.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.697062 11718 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:03:03.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.697227 11718 kserver.cc:163] Server-wide thread pool size limit: 3276
14:03:03.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.697500 11718 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:03:03.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.698109 11743 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
14:03:03.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.699011 11718 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:03:03.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.699084 11718 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s	user 0.000s	sys 0.000s
14:03:03.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.699126 11718 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:03:03.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.699775 11718 ts_tablet_manager.cc:616] Registered 1 tablets
14:03:03.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.699834 11718 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s	user 0.000s	sys 0.000s
14:03:03.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.699898 11743 tablet_bootstrap.cc:492] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196: Bootstrap starting.
14:03:03.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.707808 11718 rpc_server.cc:307] RPC server started. Bound to: 127.10.89.66:34685
14:03:03.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.707906 11863 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.89.66:34685 every 8 connection(s)
14:03:03.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.708256 11718 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/data/info.pb
14:03:03.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.709803 11743 log.cc:826] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196: Log is configured to *not* fsync() on all Append() calls
14:03:03.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.712819 11743 tablet_bootstrap.cc:492] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=6 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:03:03.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.712935 10597 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskhBFXjG/build/release/bin/kudu as pid 11718
14:03:03.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.713136 11743 tablet_bootstrap.cc:492] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196: Bootstrap complete.
14:03:03.713 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:03:03.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.714570 11743 ts_tablet_manager.cc:1403] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196: Time spent bootstrapping tablet: real 0.015s	user 0.003s	sys 0.010s
14:03:03.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.715173 11865 heartbeater.cc:344] Connected to a master server at 127.10.89.126:36189
14:03:03.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.715260 11865 heartbeater.cc:461] Registering TS with master...
14:03:03.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.715469 11865 heartbeater.cc:507] Master 127.10.89.126:36189 requested a full tablet report, sending...
14:03:03.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.715958 11866 heartbeater.cc:344] Connected to a master server at 127.10.89.125:42399
14:03:03.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.716018 11866 heartbeater.cc:461] Registering TS with master...
14:03:03.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.716060 10643 ts_manager.cc:194] Re-registered known tserver with Master: 7bf0930626894cff94471c02ef2b9196 (127.10.89.66:34685)
14:03:03.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.716110 11866 heartbeater.cc:507] Master 127.10.89.125:42399 requested a full tablet report, sending...
14:03:03.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.715929 11743 raft_consensus.cc:359] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:03.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.716207 11743 raft_consensus.cc:740] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bf0930626894cff94471c02ef2b9196, State: Initialized, Role: FOLLOWER
14:03:03.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.716338 11743 consensus_queue.cc:260] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 3.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:03.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.716619 11743 ts_tablet_manager.cc:1434] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.003s
14:03:03.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.717267 10732 ts_manager.cc:194] Re-registered known tserver with Master: 7bf0930626894cff94471c02ef2b9196 (127.10.89.66:34685)
14:03:03.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.717810 10732 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.10.89.66:45149
14:03:03.718 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 9 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823, tablet=28da754494014b9f85434e4ebbbef9d7, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:03:03.719 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.10.89.65:36825
14:03:03.720 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

14:03:03.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.718145 11864 heartbeater.cc:344] Connected to a master server at 127.10.89.124:42407
14:03:03.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.718212 11864 heartbeater.cc:461] Registering TS with master...
14:03:03.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.718307 11864 heartbeater.cc:507] Master 127.10.89.124:42407 requested a full tablet report, sending...
14:03:03.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.719739 10813 ts_manager.cc:194] Re-registered known tserver with Master: 7bf0930626894cff94471c02ef2b9196 (127.10.89.66:34685)
14:03:03.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.720026 11866 heartbeater.cc:499] Master 127.10.89.125:42399 was elected leader, sending a full tablet report...
14:03:03.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:03.720412 10597 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskhBFXjG/build/release/bin/kudu with pid 10924
14:03:03.733 [DEBUG - kudu-nio-1] (Connection.java:239) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] handling channelInactive
14:03:03.733 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:03:03.733 [DEBUG - kudu-nio-1] (Connection.java:709) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] cleaning up while in state READY due to: connection closed
14:03:03.734 [DEBUG - Time-limited test] (Connection.java:748) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] connecting to peer
14:03:03.735 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:03.736 [DEBUG - kudu-nio-2] (Connection.java:709) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] cleaning up while in state CONNECTING due to: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:03.736 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:03.747 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:03.748 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723),7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685),d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)[L]]}]
14:03:03.748 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=2, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=14), Traces: [0ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [2ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [2ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [13ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@649528059(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@55a15aa7 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@21bdaf3e -> wakeup thread Time-limited test)) after lookup
14:03:03.748 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] connecting to peer
14:03:03.750 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:03.750 [DEBUG - kudu-nio-3] (Connection.java:709) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] cleaning up while in state CONNECTING due to: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:03.750 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:03.767 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:03.768 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723),7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685),d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)[L]]}]
14:03:03.768 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=4, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=35), Traces: [0ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [2ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [2ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [13ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [14ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [16ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [16ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [30ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [33ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@649528059(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@55a15aa7 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2111129986 after retry RPC@836623705), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@21bdaf3e -> wakeup thread Time-limited test -> (continuation of Deferred@2111129986 after retry RPC@836623705))) after lookup
14:03:03.769 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] connecting to peer
14:03:03.770 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:03.770 [DEBUG - kudu-nio-4] (Connection.java:709) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] cleaning up while in state CONNECTING due to: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:03.771 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:03.788 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:03.788 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723),7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685),d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)[L]]}]
14:03:03.788 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=6, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=54), Traces: [0ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [2ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [2ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [13ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [14ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [16ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [16ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [30ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [33ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [35ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [36ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [37ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [50ms] refreshing cache from master, [50ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [53ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@649528059(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@55a15aa7 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2111129986 after retry RPC@836623705) -> (continuation of Deferred@195946427 after retry RPC@1422247042), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@21bdaf3e -> wakeup thread Time-limited test -> (continuation of Deferred@2111129986 after retry RPC@836623705) -> (continuation of Deferred@195946427 after retry RPC@1422247042))) after lookup
14:03:03.789 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] connecting to peer
14:03:03.790 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:03.790 [DEBUG - kudu-nio-5] (Connection.java:709) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] cleaning up while in state CONNECTING due to: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:03.791 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:03.827 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:03.828 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723),7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685),d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)[L]]}]
14:03:03.828 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=8, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=94), Traces: [0ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [2ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [2ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [13ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [14ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [16ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [16ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [30ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [33ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [35ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [36ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [37ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [50ms] refreshing cache from master, [50ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [53ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [55ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [57ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [57ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [90ms] refreshing cache from master, [90ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [93ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@649528059(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@55a15aa7 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2111129986 after retry RPC@836623705) -> (continuation of Deferred@195946427 after retry RPC@1422247042) -> (continuation of Deferred@1430444947 after retry RPC@802808941), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@21bdaf3e -> wakeup thread Time-limited test -> (continuation of Deferred@2111129986 after retry RPC@836623705) -> (continuation of Deferred@195946427 after retry RPC@1422247042) -> (continuation of Deferred@1430444947 after retry RPC@802808941))) after lookup
14:03:03.829 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] connecting to peer
14:03:03.830 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:03.830 [DEBUG - kudu-nio-6] (Connection.java:709) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] cleaning up while in state CONNECTING due to: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:03.831 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:03.927 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:03.927 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723),7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685),d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)[L]]}]
14:03:03.928 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=10, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=194), Traces: [0ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [2ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [2ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [13ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [14ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [16ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [16ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [30ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [33ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [35ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [36ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [37ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [50ms] refreshing cache from master, [50ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [53ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [55ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [57ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [57ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [90ms] refreshing cache from master, [90ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [93ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [95ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [97ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [97ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [190ms] refreshing cache from master, [191ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [193ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@649528059(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@55a15aa7 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2111129986 after retry RPC@836623705) -> (continuation of Deferred@195946427 after retry RPC@1422247042) -> (continuation of Deferred@1430444947 after retry RPC@802808941) -> (continuation of Deferred@658206218 after retry RPC@1193348108), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@21bdaf3e -> wakeup thread Time-limited test -> (continuation of Deferred@2111129986 after retry RPC@836623705) -> (continuation of Deferred@195946427 after retry RPC@1422247042) -> (continuation of Deferred@1430444947 after retry RPC@802808941) -> (continuation of Deferred@658206218 after retry RPC@1193348108))) after lookup
14:03:03.928 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] connecting to peer
14:03:03.929 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:03.930 [DEBUG - kudu-nio-7] (Connection.java:709) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] cleaning up while in state CONNECTING due to: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:03.930 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:05.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.085213 11878 raft_consensus.cc:493] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d8b415ec6a9a4671b34e29775c855962)
14:03:05.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.085317 11878 raft_consensus.cc:515] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:05.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.085697 11878 leader_election.cc:290] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8b415ec6a9a4671b34e29775c855962 (127.10.89.65:36825), 7bf0930626894cff94471c02ef2b9196 (127.10.89.66:34685)
14:03:05.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:05.086184 11536 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.89.65:36825: connect: Connection refused (error 111)
14:03:05.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:05.086643 11536 leader_election.cc:336] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d8b415ec6a9a4671b34e29775c855962 (127.10.89.65:36825): Network error: Client connection negotiation failed: client connection to 127.10.89.65:36825: connect: Connection refused (error 111)
14:03:05.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.089072 11814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28da754494014b9f85434e4ebbbef9d7" candidate_uuid: "751f3be0b90f4a1d86af88ca13d42016" candidate_term: 5 candidate_status { last_received { term: 4 index: 12 } } ignore_live_leader: false dest_uuid: "7bf0930626894cff94471c02ef2b9196" is_pre_election: true
14:03:05.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.089227 11814 raft_consensus.cc:2468] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 751f3be0b90f4a1d86af88ca13d42016 in term 3.
14:03:05.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.089483 11533 leader_election.cc:304] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 751f3be0b90f4a1d86af88ca13d42016, 7bf0930626894cff94471c02ef2b9196; no voters: d8b415ec6a9a4671b34e29775c855962
14:03:05.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.089624 11878 raft_consensus.cc:2804] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 4 FOLLOWER]: Leader pre-election won for term 5
14:03:05.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.089676 11878 raft_consensus.cc:493] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d8b415ec6a9a4671b34e29775c855962)
14:03:05.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.089701 11878 raft_consensus.cc:3060] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 4 FOLLOWER]: Advancing to term 5
14:03:05.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.090405 11878 raft_consensus.cc:515] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:05.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.090569 11878 leader_election.cc:290] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [CANDIDATE]: Term 5 election: Requested vote from peers d8b415ec6a9a4671b34e29775c855962 (127.10.89.65:36825), 7bf0930626894cff94471c02ef2b9196 (127.10.89.66:34685)
14:03:05.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.090804 11814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28da754494014b9f85434e4ebbbef9d7" candidate_uuid: "751f3be0b90f4a1d86af88ca13d42016" candidate_term: 5 candidate_status { last_received { term: 4 index: 12 } } ignore_live_leader: false dest_uuid: "7bf0930626894cff94471c02ef2b9196"
14:03:05.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.090900 11814 raft_consensus.cc:3060] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 3 FOLLOWER]: Advancing to term 5
14:03:05.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:05.091174 11536 leader_election.cc:336] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer d8b415ec6a9a4671b34e29775c855962 (127.10.89.65:36825): Network error: Client connection negotiation failed: client connection to 127.10.89.65:36825: connect: Connection refused (error 111)
14:03:05.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.092062 11814 raft_consensus.cc:2468] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 751f3be0b90f4a1d86af88ca13d42016 in term 5.
14:03:05.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.092371 11533 leader_election.cc:304] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 751f3be0b90f4a1d86af88ca13d42016, 7bf0930626894cff94471c02ef2b9196; no voters: d8b415ec6a9a4671b34e29775c855962
14:03:05.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.092484 11878 raft_consensus.cc:2804] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 5 FOLLOWER]: Leader election won for term 5
14:03:05.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.092659 11878 raft_consensus.cc:697] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 5 LEADER]: Becoming Leader. State: Replica: 751f3be0b90f4a1d86af88ca13d42016, State: Running, Role: LEADER
14:03:05.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.092790 11878 consensus_queue.cc:237] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 4.12, Last appended by leader: 12, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:05.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.093883 10732 catalog_manager.cc:5719] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 reported cstate change: term changed from 4 to 5, leader changed from d8b415ec6a9a4671b34e29775c855962 (127.10.89.65) to 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67). New cstate: current_term: 5 leader_uuid: "751f3be0b90f4a1d86af88ca13d42016" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } health_report { overall_health: UNKNOWN } } }
14:03:05.168 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:05.168 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)[L],7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685),d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)]}]
14:03:05.168 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=12, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=1435), Traces: [0ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [2ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [2ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [11ms] refreshing cache from master, [11ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [13ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [14ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [16ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [16ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [30ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [33ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [35ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [36ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [37ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [50ms] refreshing cache from master, [50ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [53ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [55ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [57ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [57ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [90ms] refreshing cache from master, [90ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [93ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [95ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [97ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [97ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [190ms] refreshing cache from master, [191ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [193ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [194ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [196ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [196ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [1431ms] refreshing cache from master, [1431ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [1433ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@649528059(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@55a15aa7 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2111129986 after retry RPC@836623705) -> (continuation of Deferred@195946427 after retry RPC@1422247042) -> (continuation of Deferred@1430444947 after retry RPC@802808941) -> (continuation of Deferred@658206218 after retry RPC@1193348108) -> (continuation of Deferred@328698799 after retry RPC@1553577149), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@21bdaf3e -> wakeup thread Time-limited test -> (continuation of Deferred@2111129986 after retry RPC@836623705) -> (continuation of Deferred@195946427 after retry RPC@1422247042) -> (continuation of Deferred@1430444947 after retry RPC@802808941) -> (continuation of Deferred@658206218 after retry RPC@1193348108) -> (continuation of Deferred@328698799 after retry RPC@1553577149))) after lookup
14:03:05.169 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] connecting to peer
14:03:05.170 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] Successfully connected to peer
14:03:05.187 [DEBUG - kudu-nio-0] (Negotiator.java:918) Authenticated connection [id: 0x5da7225a, L:/127.0.0.1:54742 - R:/127.10.89.67:46723] using TOKEN/null
14:03:05.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.190338 11814 raft_consensus.cc:1275] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 5 FOLLOWER]: Refusing update from remote peer 751f3be0b90f4a1d86af88ca13d42016: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 5 index: 14. (index mismatch)
14:03:05.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.190680 11878 consensus_queue.cc:1048] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 8, Time since last communication: 0.000s
14:03:05.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:05.191017 11536 consensus_peers.cc:597] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 -> Peer d8b415ec6a9a4671b34e29775c855962 (127.10.89.65:36825): Couldn't send request to peer d8b415ec6a9a4671b34e29775c855962. Status: Network error: Client connection negotiation failed: client connection to 127.10.89.65:36825: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:03:05.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.193698 11900 mvcc.cc:204] Tried to move back new op lower bound from 7317327811337072640 to 7317327810941042688. Current Snapshot: MvccSnapshot[applied={T|T < 7317327811337072640}]
14:03:05.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.194418 11902 mvcc.cc:204] Tried to move back new op lower bound from 7317327811337072640 to 7317327810941042688. Current Snapshot: MvccSnapshot[applied={T|T < 7317327811337072640}]
14:03:05.207 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.10.89.65:36825
14:03:05.207 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

14:03:05.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.208113 10597 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskhBFXjG/build/release/bin/kudu
14:03:05.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskhBFXjG/build/release/bin/kudu
14:03:05.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/wal
14:03:05.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/data
14:03:05.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:03:05.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:03:05.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:03:05.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:03:05.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:03:05.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:03:05.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/logs
14:03:05.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/data/info.pb
14:03:05.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:03:05.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:03:05.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:03:05.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:03:05.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:03:05.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:03:05.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:03:05.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:03:05.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:03:05.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:03:05.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.89.65:36825
14:03:05.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.89.65
14:03:05.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.89.126:36189,127.10.89.125:42399,127.10.89.124:42407
14:03:05.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43475
14:03:05.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.89.65
14:03:05.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.89.84:33487
14:03:05.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:03:05.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:03:05.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:03:05.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:05.291478 11909 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:03:05.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:05.291653 11909 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:03:05.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:05.291674 11909 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:03:05.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:05.293157 11909 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:03:05.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:05.293205 11909 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.89.65
14:03:05.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.294905 11909 tablet_server_runner.cc:78] Tablet server non-default flags:
14:03:05.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/logs
14:03:05.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:03:05.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:03:05.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:03:05.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.89.84:33487
14:03:05.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:03:05.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:03:05.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/data
14:03:05.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/wal
14:03:05.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:03:05.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:03:05.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.89.65:36825
14:03:05.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:03:05.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:03:05.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:03:05.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/data/info.pb
14:03:05.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.89.65
14:03:05.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43475
14:03:05.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.89.126:36189,127.10.89.125:42399,127.10.89.124:42407
14:03:05.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:03:05.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.11909
14:03:05.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:03:05.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:03:05.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:03:05.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:03:05.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.89.65
14:03:05.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:03:05.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:03:05.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:03:05.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 4bcd631c449d3c6649730ca6e07660c8168f0d44
14:03:05.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
14:03:05.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Aug 2026 13:43:13 UTC on e7f111948823
14:03:05.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 13289
14:03:05.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.295116 11909 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:03:05.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.295300 11909 file_cache.cc:504] Constructed file cache file cache with capacity 419430
14:03:05.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:05.298306 11914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:03:05.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:05.298295 11917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:03:05.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.298503 11909 server_base.cc:1061] running on GCE node
14:03:05.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:05.298292 11915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:03:05.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.299535 11909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:03:05.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.299779 11909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:03:05.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.300937 11909 hybrid_clock.cc:648] HybridClock initialized: now 1786456985300907 us; error 44 us; skew 500 ppm
14:03:05.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.302094 11909 webserver.cc:533] Webserver started at http://127.10.89.65:43475/ using document root <none> and password file <none>
14:03:05.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.302277 11909 fs_manager.cc:362] Metadata directory not provided
14:03:05.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.302323 11909 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:03:05.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.303548 11909 fs_manager.cc:714] Time spent opening directory manager: real 0.001s	user 0.001s	sys 0.000s
14:03:05.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.304186 11926 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:03:05.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.304376 11909 fs_manager.cc:730] Time spent opening block manager: real 0.001s	user 0.001s	sys 0.000s
14:03:05.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.304433 11909 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/data,/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/wal
14:03:05.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d8b415ec6a9a4671b34e29775c855962"
14:03:05.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 14:02:53 on dist-test-slave-g350"
14:03:05.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.304739 11909 fs_report.cc:389] FS layout report
14:03:05.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:03:05.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/wal
14:03:05.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/wal
14:03:05.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/data/data
14:03:05.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:03:05.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:03:05.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:03:05.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:03:05.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:03:05.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:03:05.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:03:05.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:03:05.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:03:05.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:03:05.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:03:05.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.325661 11909 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:03:05.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.325978 11909 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:03:05.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.326068 11909 kserver.cc:163] Server-wide thread pool size limit: 3276
14:03:05.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.326265 11909 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:03:05.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.326912 11935 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
14:03:05.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.327878 11909 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:03:05.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.327948 11909 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s	user 0.000s	sys 0.000s
14:03:05.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.327987 11909 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:03:05.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.328702 11909 ts_tablet_manager.cc:616] Registered 1 tablets
14:03:05.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.328758 11909 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s	user 0.000s	sys 0.000s
14:03:05.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.328812 11935 tablet_bootstrap.cc:492] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962: Bootstrap starting.
14:03:05.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.336632 11909 rpc_server.cc:307] RPC server started. Bound to: 127.10.89.65:36825
14:03:05.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.336673 12052 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.89.65:36825 every 8 connection(s)
14:03:05.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.337029 11909 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/data/info.pb
14:03:05.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.340214 11935 log.cc:826] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962: Log is configured to *not* fsync() on all Append() calls
14:03:05.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.343920 10597 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskhBFXjG/build/release/bin/kudu as pid 11909
14:03:05.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.343927 11935 tablet_bootstrap.cc:492] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=9 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:03:05.344 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:03:05.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.344297 11935 tablet_bootstrap.cc:492] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962: Bootstrap complete.
14:03:05.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.345696 11935 ts_tablet_manager.cc:1403] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962: Time spent bootstrapping tablet: real 0.017s	user 0.002s	sys 0.014s
14:03:05.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.346997 11935 raft_consensus.cc:359] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:05.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.347334 11935 raft_consensus.cc:740] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b415ec6a9a4671b34e29775c855962, State: Initialized, Role: FOLLOWER
14:03:05.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.347482 11935 consensus_queue.cc:260] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 4.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:05.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.347806 11935 ts_tablet_manager.cc:1434] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
14:03:05.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.349089 12055 heartbeater.cc:344] Connected to a master server at 127.10.89.124:42407
14:03:05.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.349191 12055 heartbeater.cc:461] Registering TS with master...
14:03:05.350 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 12 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823, tablet=28da754494014b9f85434e4ebbbef9d7, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:03:05.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.349401 12055 heartbeater.cc:507] Master 127.10.89.124:42407 requested a full tablet report, sending...
14:03:05.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.351723 12057 heartbeater.cc:344] Connected to a master server at 127.10.89.125:42399
14:03:05.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.351783 12057 heartbeater.cc:461] Registering TS with master...
14:03:05.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.351904 12057 heartbeater.cc:507] Master 127.10.89.125:42399 requested a full tablet report, sending...
14:03:05.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.352205 10732 ts_manager.cc:194] Re-registered known tserver with Master: d8b415ec6a9a4671b34e29775c855962 (127.10.89.65:36825)
14:03:05.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.352634 10732 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.10.89.65:37617
14:03:05.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.352896 10814 ts_manager.cc:194] Re-registered known tserver with Master: d8b415ec6a9a4671b34e29775c855962 (127.10.89.65:36825)
14:03:05.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.353420 12057 heartbeater.cc:499] Master 127.10.89.125:42399 was elected leader, sending a full tablet report...
14:03:05.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.357191 12056 heartbeater.cc:344] Connected to a master server at 127.10.89.126:36189
14:03:05.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.357321 12056 heartbeater.cc:461] Registering TS with master...
14:03:05.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.357386 12056 heartbeater.cc:507] Master 127.10.89.126:36189 requested a full tablet report, sending...
14:03:05.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.357795 10643 ts_manager.cc:194] Re-registered known tserver with Master: d8b415ec6a9a4671b34e29775c855962 (127.10.89.65:36825)
14:03:05.365 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.10.89.67:46723
14:03:05.366 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

14:03:05.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:05.366596 10597 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskhBFXjG/build/release/bin/kudu with pid 11516
14:03:05.379 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:03:05.379 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] handling channelInactive
14:03:05.379 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] cleaning up while in state READY due to: connection closed
14:03:05.380 [DEBUG - Time-limited test] (Connection.java:748) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] connecting to peer
14:03:05.381 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:05.381 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] cleaning up while in state CONNECTING due to: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:05.381 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:05.387 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:05.388 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)[L],7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685),d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)]}]
14:03:05.388 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=2, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=9), Traces: [0ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [2ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [2ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [8ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@551001968(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@2a2df458 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@269b4947 -> wakeup thread Time-limited test)) after lookup
14:03:05.389 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] connecting to peer
14:03:05.391 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:05.391 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] cleaning up while in state CONNECTING due to: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:05.391 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:05.407 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:05.408 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)[L],7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685),d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)]}]
14:03:05.408 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=4, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=29), Traces: [0ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [2ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [2ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [8ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [10ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [12ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [12ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [25ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [28ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@551001968(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@2a2df458 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@156092198 after retry RPC@1969255993), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@269b4947 -> wakeup thread Time-limited test -> (continuation of Deferred@156092198 after retry RPC@1969255993))) after lookup
14:03:05.409 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] connecting to peer
14:03:05.410 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:05.410 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] cleaning up while in state CONNECTING due to: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:05.410 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:05.427 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:05.427 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)[L],7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685),d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)]}]
14:03:05.427 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=6, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=48), Traces: [0ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [2ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [2ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [8ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [10ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [12ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [12ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [25ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [28ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [29ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [31ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [31ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [45ms] refreshing cache from master, [45ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [47ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@551001968(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@2a2df458 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@156092198 after retry RPC@1969255993) -> (continuation of Deferred@261484471 after retry RPC@2033357250), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@269b4947 -> wakeup thread Time-limited test -> (continuation of Deferred@156092198 after retry RPC@1969255993) -> (continuation of Deferred@261484471 after retry RPC@2033357250))) after lookup
14:03:05.429 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] connecting to peer
14:03:05.430 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:05.430 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] cleaning up while in state CONNECTING due to: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:05.430 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:05.527 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:05.527 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)[L],7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685),d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)]}]
14:03:05.528 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=8, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=148), Traces: [0ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [2ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [2ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [8ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [10ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [12ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [12ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [25ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [28ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [29ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [31ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [31ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [45ms] refreshing cache from master, [45ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [47ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [49ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [51ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [51ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [145ms] refreshing cache from master, [145ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [148ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@551001968(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@2a2df458 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@156092198 after retry RPC@1969255993) -> (continuation of Deferred@261484471 after retry RPC@2033357250) -> (continuation of Deferred@1591687109 after retry RPC@856559965), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@269b4947 -> wakeup thread Time-limited test -> (continuation of Deferred@156092198 after retry RPC@1969255993) -> (continuation of Deferred@261484471 after retry RPC@2033357250) -> (continuation of Deferred@1591687109 after retry RPC@856559965))) after lookup
14:03:05.528 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] connecting to peer
14:03:05.529 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:05.529 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] cleaning up while in state CONNECTING due to: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:05.530 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:05.727 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:05.728 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)[L],7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685),d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)]}]
14:03:05.728 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=10, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=348), Traces: [0ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [2ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [2ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [8ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [10ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [12ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [12ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [25ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [28ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [29ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [31ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [31ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [45ms] refreshing cache from master, [45ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [47ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [49ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [51ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [51ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [145ms] refreshing cache from master, [145ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [148ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [149ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [151ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [151ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [345ms] refreshing cache from master, [346ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [348ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@551001968(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@2a2df458 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@156092198 after retry RPC@1969255993) -> (continuation of Deferred@261484471 after retry RPC@2033357250) -> (continuation of Deferred@1591687109 after retry RPC@856559965) -> (continuation of Deferred@2108937108 after retry RPC@1056219526), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@269b4947 -> wakeup thread Time-limited test -> (continuation of Deferred@156092198 after retry RPC@1969255993) -> (continuation of Deferred@261484471 after retry RPC@2033357250) -> (continuation of Deferred@1591687109 after retry RPC@856559965) -> (continuation of Deferred@2108937108 after retry RPC@1056219526))) after lookup
14:03:05.729 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] connecting to peer
14:03:05.730 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:05.730 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] cleaning up while in state CONNECTING due to: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:05.730 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:06.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:06.706643 12073 raft_consensus.cc:493] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 751f3be0b90f4a1d86af88ca13d42016)
14:03:06.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:06.706751 12073 raft_consensus.cc:515] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:06.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:06.707026 12073 leader_election.cc:290] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723), d8b415ec6a9a4671b34e29775c855962 (127.10.89.65:36825)
14:03:06.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:06.707595 11740 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.89.67:46723: connect: Connection refused (error 111)
14:03:06.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:06.708266 11740 leader_election.cc:336] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723): Network error: Client connection negotiation failed: client connection to 127.10.89.67:46723: connect: Connection refused (error 111)
14:03:06.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:06.710992 12004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28da754494014b9f85434e4ebbbef9d7" candidate_uuid: "7bf0930626894cff94471c02ef2b9196" candidate_term: 6 candidate_status { last_received { term: 5 index: 16 } } ignore_live_leader: false dest_uuid: "d8b415ec6a9a4671b34e29775c855962" is_pre_election: true
14:03:06.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:06.711172 12004 raft_consensus.cc:2468] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bf0930626894cff94471c02ef2b9196 in term 4.
14:03:06.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:06.711510 11740 leader_election.cc:304] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7bf0930626894cff94471c02ef2b9196, d8b415ec6a9a4671b34e29775c855962; no voters: 751f3be0b90f4a1d86af88ca13d42016
14:03:06.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:06.711637 12073 raft_consensus.cc:2804] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 5 FOLLOWER]: Leader pre-election won for term 6
14:03:06.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:06.711712 12073 raft_consensus.cc:493] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 751f3be0b90f4a1d86af88ca13d42016)
14:03:06.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:06.711738 12073 raft_consensus.cc:3060] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 5 FOLLOWER]: Advancing to term 6
14:03:06.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:06.712589 12073 raft_consensus.cc:515] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:06.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:06.712774 12073 leader_election.cc:290] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [CANDIDATE]: Term 6 election: Requested vote from peers 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723), d8b415ec6a9a4671b34e29775c855962 (127.10.89.65:36825)
14:03:06.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:06.713101 12004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28da754494014b9f85434e4ebbbef9d7" candidate_uuid: "7bf0930626894cff94471c02ef2b9196" candidate_term: 6 candidate_status { last_received { term: 5 index: 16 } } ignore_live_leader: false dest_uuid: "d8b415ec6a9a4671b34e29775c855962"
14:03:06.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:06.713220 12004 raft_consensus.cc:3060] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 4 FOLLOWER]: Advancing to term 6
14:03:06.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:06.713335 11740 leader_election.cc:336] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723): Network error: Client connection negotiation failed: client connection to 127.10.89.67:46723: connect: Connection refused (error 111)
14:03:06.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:06.714319 12004 raft_consensus.cc:2468] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bf0930626894cff94471c02ef2b9196 in term 6.
14:03:06.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:06.714522 11740 leader_election.cc:304] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7bf0930626894cff94471c02ef2b9196, d8b415ec6a9a4671b34e29775c855962; no voters: 751f3be0b90f4a1d86af88ca13d42016
14:03:06.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:06.714633 12073 raft_consensus.cc:2804] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 6 FOLLOWER]: Leader election won for term 6
14:03:06.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:06.714779 12073 raft_consensus.cc:697] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 6 LEADER]: Becoming Leader. State: Replica: 7bf0930626894cff94471c02ef2b9196, State: Running, Role: LEADER
14:03:06.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:06.714849 12073 consensus_queue.cc:237] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 5.16, Last appended by leader: 16, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:06.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:06.715615 10732 catalog_manager.cc:5719] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 reported cstate change: term changed from 5 to 6, leader changed from 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67) to 7bf0930626894cff94471c02ef2b9196 (127.10.89.66). New cstate: current_term: 6 leader_uuid: "7bf0930626894cff94471c02ef2b9196" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } health_report { overall_health: HEALTHY } } }
14:03:07.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.122392 12004 raft_consensus.cc:1275] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 6 FOLLOWER]: Refusing update from remote peer 7bf0930626894cff94471c02ef2b9196: Log matching property violated. Preceding OpId in replica: term: 4 index: 12. Preceding OpId from leader: term: 6 index: 17. (index mismatch)
14:03:07.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.122922 12073 consensus_queue.cc:1048] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 12, Time since last communication: 0.000s
14:03:07.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:07.126533 11740 consensus_peers.cc:597] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 -> Peer 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723): Couldn't send request to peer 751f3be0b90f4a1d86af88ca13d42016. Status: Network error: Client connection negotiation failed: client connection to 127.10.89.67:46723: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:03:07.647 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:07.648 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723),7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)[L],d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)]}]
14:03:07.648 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=12, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=2269), Traces: [0ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [2ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [2ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [8ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [10ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [12ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [12ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [25ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [28ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [29ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [31ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [31ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [45ms] refreshing cache from master, [45ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [47ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [49ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [51ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [51ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [145ms] refreshing cache from master, [145ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [148ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [149ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [151ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [151ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [345ms] refreshing cache from master, [346ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [348ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [349ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [351ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [351ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [2266ms] refreshing cache from master, [2266ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [2268ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@551001968(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@2a2df458 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@156092198 after retry RPC@1969255993) -> (continuation of Deferred@261484471 after retry RPC@2033357250) -> (continuation of Deferred@1591687109 after retry RPC@856559965) -> (continuation of Deferred@2108937108 after retry RPC@1056219526) -> (continuation of Deferred@1340235854 after retry RPC@1409351845), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@269b4947 -> wakeup thread Time-limited test -> (continuation of Deferred@156092198 after retry RPC@1969255993) -> (continuation of Deferred@261484471 after retry RPC@2033357250) -> (continuation of Deferred@1591687109 after retry RPC@856559965) -> (continuation of Deferred@2108937108 after retry RPC@1056219526) -> (continuation of Deferred@1340235854 after retry RPC@1409351845))) after lookup
14:03:07.649 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] connecting to peer
14:03:07.650 [DEBUG - kudu-nio-7] (Connection.java:757) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] Successfully connected to peer
14:03:07.681 [DEBUG - kudu-nio-7] (Negotiator.java:918) Authenticated connection [id: 0x08e1a606, L:/127.0.0.1:50806 - R:/127.10.89.66:34685] using TOKEN/null
14:03:07.696 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.10.89.67:46723
14:03:07.696 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

14:03:07.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.697276 10597 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskhBFXjG/build/release/bin/kudu
14:03:07.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskhBFXjG/build/release/bin/kudu
14:03:07.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/wal
14:03:07.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/data
14:03:07.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:03:07.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:03:07.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:03:07.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:03:07.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:03:07.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:03:07.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/logs
14:03:07.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/data/info.pb
14:03:07.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:03:07.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:03:07.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:03:07.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:03:07.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:03:07.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:03:07.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:03:07.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:03:07.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:03:07.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:03:07.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.89.67:46723
14:03:07.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.89.67
14:03:07.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.89.126:36189,127.10.89.125:42399,127.10.89.124:42407
14:03:07.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34339
14:03:07.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.89.67
14:03:07.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.89.84:33487
14:03:07.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:03:07.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:03:07.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:03:07.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:07.785241 12116 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:03:07.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:07.785445 12116 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:03:07.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:07.785467 12116 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:03:07.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:07.787242 12116 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:03:07.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:07.787325 12116 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.89.67
14:03:07.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.788909 12116 tablet_server_runner.cc:78] Tablet server non-default flags:
14:03:07.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/logs
14:03:07.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:03:07.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:03:07.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:03:07.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.89.84:33487
14:03:07.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:03:07.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:03:07.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/data
14:03:07.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/wal
14:03:07.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:03:07.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:03:07.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.89.67:46723
14:03:07.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:03:07.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:03:07.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:03:07.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/data/info.pb
14:03:07.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.89.67
14:03:07.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34339
14:03:07.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.89.126:36189,127.10.89.125:42399,127.10.89.124:42407
14:03:07.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:03:07.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.12116
14:03:07.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:03:07.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:03:07.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:03:07.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:03:07.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.89.67
14:03:07.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:03:07.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:03:07.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:03:07.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 4bcd631c449d3c6649730ca6e07660c8168f0d44
14:03:07.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
14:03:07.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Aug 2026 13:43:13 UTC on e7f111948823
14:03:07.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 13289
14:03:07.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.789105 12116 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:03:07.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.789415 12116 file_cache.cc:504] Constructed file cache file cache with capacity 419430
14:03:07.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:07.792234 12121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:03:07.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.792475 12116 server_base.cc:1061] running on GCE node
14:03:07.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:07.792263 12124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:03:07.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:07.792268 12122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:03:07.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.792922 12116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:03:07.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.793161 12116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:03:07.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.794330 12116 hybrid_clock.cc:648] HybridClock initialized: now 1786456987794297 us; error 48 us; skew 500 ppm
14:03:07.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.795764 12116 webserver.cc:533] Webserver started at http://127.10.89.67:34339/ using document root <none> and password file <none>
14:03:07.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.796027 12116 fs_manager.cc:362] Metadata directory not provided
14:03:07.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.796118 12116 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:03:07.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.797505 12116 fs_manager.cc:714] Time spent opening directory manager: real 0.001s	user 0.001s	sys 0.000s
14:03:07.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.798234 12132 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:03:07.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.798462 12116 fs_manager.cc:730] Time spent opening block manager: real 0.000s	user 0.000s	sys 0.001s
14:03:07.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.798532 12116 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/data,/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/wal
14:03:07.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "751f3be0b90f4a1d86af88ca13d42016"
14:03:07.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 14:02:53 on dist-test-slave-g350"
14:03:07.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.798852 12116 fs_report.cc:389] FS layout report
14:03:07.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:03:07.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/wal
14:03:07.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/wal
14:03:07.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/data/data
14:03:07.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:03:07.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:03:07.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:03:07.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:03:07.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:03:07.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:03:07.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:03:07.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:03:07.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:03:07.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:03:07.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:03:07.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.815532 12116 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:03:07.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.815950 12116 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:03:07.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.816088 12116 kserver.cc:163] Server-wide thread pool size limit: 3276
14:03:07.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.816365 12116 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:03:07.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.816954 12139 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
14:03:07.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.817920 12116 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:03:07.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.817994 12116 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s	user 0.000s	sys 0.000s
14:03:07.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.818032 12116 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:03:07.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.818650 12116 ts_tablet_manager.cc:616] Registered 1 tablets
14:03:07.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.818712 12116 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s	user 0.000s	sys 0.000s
14:03:07.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.818771 12139 tablet_bootstrap.cc:492] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016: Bootstrap starting.
14:03:07.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.825886 12116 rpc_server.cc:307] RPC server started. Bound to: 127.10.89.67:46723
14:03:07.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.825974 12267 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.89.67:46723 every 8 connection(s)
14:03:07.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.826355 12116 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/data/info.pb
14:03:07.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.830662 12139 log.cc:826] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016: Log is configured to *not* fsync() on all Append() calls
14:03:07.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.832702 12268 heartbeater.cc:344] Connected to a master server at 127.10.89.124:42407
14:03:07.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.832840 12268 heartbeater.cc:461] Registering TS with master...
14:03:07.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.833040 12268 heartbeater.cc:507] Master 127.10.89.124:42407 requested a full tablet report, sending...
14:03:07.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.833799 10813 ts_manager.cc:194] Re-registered known tserver with Master: 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723)
14:03:07.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.833817 10597 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskhBFXjG/build/release/bin/kudu as pid 12116
14:03:07.834 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:03:07.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.834563 12269 heartbeater.cc:344] Connected to a master server at 127.10.89.126:36189
14:03:07.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.834627 12269 heartbeater.cc:461] Registering TS with master...
14:03:07.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.834723 12269 heartbeater.cc:507] Master 127.10.89.126:36189 requested a full tablet report, sending...
14:03:07.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.834801 12139 tablet_bootstrap.cc:492] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=12 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:03:07.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.835145 12139 tablet_bootstrap.cc:492] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016: Bootstrap complete.
14:03:07.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.835593 10644 ts_manager.cc:194] Re-registered known tserver with Master: 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723)
14:03:07.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.836321 12270 heartbeater.cc:344] Connected to a master server at 127.10.89.125:42399
14:03:07.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.836385 12270 heartbeater.cc:461] Registering TS with master...
14:03:07.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.836506 12270 heartbeater.cc:507] Master 127.10.89.125:42399 requested a full tablet report, sending...
14:03:07.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.837018 12139 ts_tablet_manager.cc:1403] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016: Time spent bootstrapping tablet: real 0.018s	user 0.002s	sys 0.014s
14:03:07.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.837082 10731 ts_manager.cc:194] Re-registered known tserver with Master: 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723)
14:03:07.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.837666 10731 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.10.89.67:54303
14:03:07.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.838405 12139 raft_consensus.cc:359] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:07.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.838706 12139 raft_consensus.cc:740] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 751f3be0b90f4a1d86af88ca13d42016, State: Initialized, Role: FOLLOWER
14:03:07.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.838832 12139 consensus_queue.cc:260] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 5.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:07.839 [DEBUG - kudu-nio-7] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 15 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823, tablet=28da754494014b9f85434e4ebbbef9d7, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:03:07.839 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.839149 12139 ts_tablet_manager.cc:1434] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
14:03:07.839 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.839303 12270 heartbeater.cc:499] Master 127.10.89.125:42399 was elected leader, sending a full tablet report...
14:03:07.840 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.10.89.66:34685
14:03:07.840 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

14:03:07.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:07.840931 10597 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskhBFXjG/build/release/bin/kudu with pid 11718
14:03:07.853 [DEBUG - kudu-nio-7] (Connection.java:239) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] handling channelInactive
14:03:07.853 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] cleaning up while in state READY due to: connection closed
14:03:07.854 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:03:07.854 [DEBUG - Time-limited test] (Connection.java:748) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] connecting to peer
14:03:07.855 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:07.856 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:07.856 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:07.867 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:07.867 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723),7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)[L],d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)]}]
14:03:07.868 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=2, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=13), Traces: [0ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [2ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [2ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [10ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [13ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@1756221913(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@34f5f041 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@54d4336 -> wakeup thread Time-limited test)) after lookup
14:03:07.868 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] connecting to peer
14:03:07.870 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:07.870 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:07.870 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:07.887 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:07.888 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723),7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)[L],d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)]}]
14:03:07.888 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=4, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=34), Traces: [0ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [2ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [2ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [10ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [13ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [14ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [16ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [16ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [33ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@1756221913(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@34f5f041 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1182653952 after retry RPC@1465773195), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@54d4336 -> wakeup thread Time-limited test -> (continuation of Deferred@1182653952 after retry RPC@1465773195))) after lookup
14:03:07.889 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] connecting to peer
14:03:07.890 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:07.890 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:07.890 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:07.927 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:07.927 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723),7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)[L],d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)]}]
14:03:07.928 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=6, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=73), Traces: [0ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [2ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [2ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [10ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [13ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [14ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [16ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [16ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [33ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [34ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [36ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [36ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [71ms] refreshing cache from master, [71ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [73ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@1756221913(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@34f5f041 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1182653952 after retry RPC@1465773195) -> (continuation of Deferred@928536333 after retry RPC@265936089), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@54d4336 -> wakeup thread Time-limited test -> (continuation of Deferred@1182653952 after retry RPC@1465773195) -> (continuation of Deferred@928536333 after retry RPC@265936089))) after lookup
14:03:07.929 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] connecting to peer
14:03:07.930 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:07.930 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:07.931 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:07.987 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:07.988 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723),7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)[L],d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)]}]
14:03:07.988 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=8, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=134), Traces: [0ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [2ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [2ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [10ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [13ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [14ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [16ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [16ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [33ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [34ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [36ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [36ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [71ms] refreshing cache from master, [71ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [73ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [74ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [77ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [77ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [130ms] refreshing cache from master, [131ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [133ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@1756221913(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@34f5f041 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1182653952 after retry RPC@1465773195) -> (continuation of Deferred@928536333 after retry RPC@265936089) -> (continuation of Deferred@632904390 after retry RPC@1690793404), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@54d4336 -> wakeup thread Time-limited test -> (continuation of Deferred@1182653952 after retry RPC@1465773195) -> (continuation of Deferred@928536333 after retry RPC@265936089) -> (continuation of Deferred@632904390 after retry RPC@1690793404))) after lookup
14:03:07.989 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] connecting to peer
14:03:07.990 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:07.990 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:07.990 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:08.447 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:08.448 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723),7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)[L],d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)]}]
14:03:08.448 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=10, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=594), Traces: [0ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [2ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [2ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [10ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [13ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [14ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [16ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [16ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [33ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [34ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [36ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [36ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [71ms] refreshing cache from master, [71ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [73ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [74ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [77ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [77ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [130ms] refreshing cache from master, [131ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [133ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [134ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [136ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [137ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [590ms] refreshing cache from master, [591ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [593ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@1756221913(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@34f5f041 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1182653952 after retry RPC@1465773195) -> (continuation of Deferred@928536333 after retry RPC@265936089) -> (continuation of Deferred@632904390 after retry RPC@1690793404) -> (continuation of Deferred@1696827227 after retry RPC@974550230), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@54d4336 -> wakeup thread Time-limited test -> (continuation of Deferred@1182653952 after retry RPC@1465773195) -> (continuation of Deferred@928536333 after retry RPC@265936089) -> (continuation of Deferred@632904390 after retry RPC@1690793404) -> (continuation of Deferred@1696827227 after retry RPC@974550230))) after lookup
14:03:08.449 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] connecting to peer
14:03:08.450 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:08.450 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:08.451 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:08.647 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:08.647 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723),7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)[L],d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)]}]
14:03:08.648 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=12, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=793), Traces: [0ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [2ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [2ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [10ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [13ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [14ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [16ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [16ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [33ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [34ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [36ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [36ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [71ms] refreshing cache from master, [71ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [73ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [74ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [77ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [77ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [130ms] refreshing cache from master, [131ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [133ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [134ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [136ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [137ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [590ms] refreshing cache from master, [591ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [593ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [595ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [596ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [597ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [790ms] refreshing cache from master, [791ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [793ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@1756221913(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@34f5f041 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1182653952 after retry RPC@1465773195) -> (continuation of Deferred@928536333 after retry RPC@265936089) -> (continuation of Deferred@632904390 after retry RPC@1690793404) -> (continuation of Deferred@1696827227 after retry RPC@974550230) -> (continuation of Deferred@1225191112 after retry RPC@1077904378), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@54d4336 -> wakeup thread Time-limited test -> (continuation of Deferred@1182653952 after retry RPC@1465773195) -> (continuation of Deferred@928536333 after retry RPC@265936089) -> (continuation of Deferred@632904390 after retry RPC@1690793404) -> (continuation of Deferred@1696827227 after retry RPC@974550230) -> (continuation of Deferred@1225191112 after retry RPC@1077904378))) after lookup
14:03:08.649 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] connecting to peer
14:03:08.650 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:08.650 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:08.651 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:09.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:09.126211 12284 raft_consensus.cc:493] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
14:03:09.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:09.126343 12284 raft_consensus.cc:515] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:09.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:09.126674 12284 leader_election.cc:290] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8b415ec6a9a4671b34e29775c855962 (127.10.89.65:36825), 7bf0930626894cff94471c02ef2b9196 (127.10.89.66:34685)
14:03:09.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:09.127141 12133 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.89.66:34685: connect: Connection refused (error 111)
14:03:09.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:09.127772 12133 leader_election.cc:336] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 7bf0930626894cff94471c02ef2b9196 (127.10.89.66:34685): Network error: Client connection negotiation failed: client connection to 127.10.89.66:34685: connect: Connection refused (error 111)
14:03:09.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:09.130260 12004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28da754494014b9f85434e4ebbbef9d7" candidate_uuid: "751f3be0b90f4a1d86af88ca13d42016" candidate_term: 6 candidate_status { last_received { term: 5 index: 16 } } ignore_live_leader: false dest_uuid: "d8b415ec6a9a4671b34e29775c855962" is_pre_election: true
14:03:09.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:09.130594 12136 leader_election.cc:304] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 751f3be0b90f4a1d86af88ca13d42016; no voters: 7bf0930626894cff94471c02ef2b9196, d8b415ec6a9a4671b34e29775c855962
14:03:09.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:09.130731 12284 raft_consensus.cc:2749] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
14:03:09.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:09.191885 12290 raft_consensus.cc:493] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7bf0930626894cff94471c02ef2b9196)
14:03:09.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:09.191995 12290 raft_consensus.cc:515] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:09.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:09.192328 12290 leader_election.cc:290] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723), 7bf0930626894cff94471c02ef2b9196 (127.10.89.66:34685)
14:03:09.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:09.192842 11927 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.89.66:34685: connect: Connection refused (error 111)
14:03:09.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:09.193343 11927 leader_election.cc:336] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 7bf0930626894cff94471c02ef2b9196 (127.10.89.66:34685): Network error: Client connection negotiation failed: client connection to 127.10.89.66:34685: connect: Connection refused (error 111)
14:03:09.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:09.195794 12214 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28da754494014b9f85434e4ebbbef9d7" candidate_uuid: "d8b415ec6a9a4671b34e29775c855962" candidate_term: 7 candidate_status { last_received { term: 6 index: 20 } } ignore_live_leader: false dest_uuid: "751f3be0b90f4a1d86af88ca13d42016" is_pre_election: true
14:03:09.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:09.195933 12214 raft_consensus.cc:2468] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8b415ec6a9a4671b34e29775c855962 in term 5.
14:03:09.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:09.196206 11932 leader_election.cc:304] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 751f3be0b90f4a1d86af88ca13d42016, d8b415ec6a9a4671b34e29775c855962; no voters: 7bf0930626894cff94471c02ef2b9196
14:03:09.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:09.196374 12290 raft_consensus.cc:2804] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 6 FOLLOWER]: Leader pre-election won for term 7
14:03:09.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:09.196427 12290 raft_consensus.cc:493] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 7bf0930626894cff94471c02ef2b9196)
14:03:09.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:09.196444 12290 raft_consensus.cc:3060] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 6 FOLLOWER]: Advancing to term 7
14:03:09.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:09.197114 12290 raft_consensus.cc:515] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:09.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:09.197307 12290 leader_election.cc:290] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [CANDIDATE]: Term 7 election: Requested vote from peers 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723), 7bf0930626894cff94471c02ef2b9196 (127.10.89.66:34685)
14:03:09.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:09.197511 12214 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28da754494014b9f85434e4ebbbef9d7" candidate_uuid: "d8b415ec6a9a4671b34e29775c855962" candidate_term: 7 candidate_status { last_received { term: 6 index: 20 } } ignore_live_leader: false dest_uuid: "751f3be0b90f4a1d86af88ca13d42016"
14:03:09.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:09.197628 12214 raft_consensus.cc:3060] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 5 FOLLOWER]: Advancing to term 7
14:03:09.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:09.198000 11927 leader_election.cc:336] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 7bf0930626894cff94471c02ef2b9196 (127.10.89.66:34685): Network error: Client connection negotiation failed: client connection to 127.10.89.66:34685: connect: Connection refused (error 111)
14:03:09.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:09.198761 12214 raft_consensus.cc:2468] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8b415ec6a9a4671b34e29775c855962 in term 7.
14:03:09.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:09.198966 11932 leader_election.cc:304] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 751f3be0b90f4a1d86af88ca13d42016, d8b415ec6a9a4671b34e29775c855962; no voters: 7bf0930626894cff94471c02ef2b9196
14:03:09.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:09.199075 12290 raft_consensus.cc:2804] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 7 FOLLOWER]: Leader election won for term 7
14:03:09.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:09.199216 12290 raft_consensus.cc:697] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 7 LEADER]: Becoming Leader. State: Replica: d8b415ec6a9a4671b34e29775c855962, State: Running, Role: LEADER
14:03:09.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:09.199290 12290 consensus_queue.cc:237] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 6.20, Last appended by leader: 20, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:09.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:09.200043 10731 catalog_manager.cc:5719] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 reported cstate change: term changed from 6 to 7, leader changed from 7bf0930626894cff94471c02ef2b9196 (127.10.89.66) to d8b415ec6a9a4671b34e29775c855962 (127.10.89.65). New cstate: current_term: 7 leader_uuid: "d8b415ec6a9a4671b34e29775c855962" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } health_report { overall_health: UNKNOWN } } }
14:03:09.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:09.699249 11927 consensus_peers.cc:597] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 -> Peer 7bf0930626894cff94471c02ef2b9196 (127.10.89.66:34685): Couldn't send request to peer 7bf0930626894cff94471c02ef2b9196. Status: Network error: Client connection negotiation failed: client connection to 127.10.89.66:34685: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:03:09.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:09.808257 12214 raft_consensus.cc:1275] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 7 FOLLOWER]: Refusing update from remote peer d8b415ec6a9a4671b34e29775c855962: Log matching property violated. Preceding OpId in replica: term: 5 index: 16. Preceding OpId from leader: term: 7 index: 21. (index mismatch)
14:03:09.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:09.808694 12290 consensus_queue.cc:1048] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [LEADER]: Connected to new peer: Peer: permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 16, Time since last communication: 0.000s
14:03:10.787 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:10.788 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723),7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685),d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)[L]]}]
14:03:10.788 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=14, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=2934), Traces: [0ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [2ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [2ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [10ms] refreshing cache from master, [10ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [13ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [14ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [16ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [16ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [33ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [34ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [36ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [36ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [71ms] refreshing cache from master, [71ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [73ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [74ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [77ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [77ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [130ms] refreshing cache from master, [131ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [133ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [134ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [136ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [137ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [590ms] refreshing cache from master, [591ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [593ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [595ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [596ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [597ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [790ms] refreshing cache from master, [791ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [793ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [794ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [797ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [797ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [2930ms] refreshing cache from master, [2931ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [2933ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@1756221913(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@34f5f041 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1182653952 after retry RPC@1465773195) -> (continuation of Deferred@928536333 after retry RPC@265936089) -> (continuation of Deferred@632904390 after retry RPC@1690793404) -> (continuation of Deferred@1696827227 after retry RPC@974550230) -> (continuation of Deferred@1225191112 after retry RPC@1077904378) -> (continuation of Deferred@1833262065 after retry RPC@1373185520), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@54d4336 -> wakeup thread Time-limited test -> (continuation of Deferred@1182653952 after retry RPC@1465773195) -> (continuation of Deferred@928536333 after retry RPC@265936089) -> (continuation of Deferred@632904390 after retry RPC@1690793404) -> (continuation of Deferred@1696827227 after retry RPC@974550230) -> (continuation of Deferred@1225191112 after retry RPC@1077904378) -> (continuation of Deferred@1833262065 after retry RPC@1373185520))) after lookup
14:03:10.789 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] connecting to peer
14:03:10.790 [DEBUG - kudu-nio-7] (Connection.java:757) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] Successfully connected to peer
14:03:10.807 [DEBUG - kudu-nio-7] (Negotiator.java:918) Authenticated connection [id: 0xc4c7eafe, L:/127.0.0.1:38422 - R:/127.10.89.65:36825] using TOKEN/null
14:03:10.824 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.10.89.66:34685
14:03:10.825 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

14:03:10.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.826138 10597 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskhBFXjG/build/release/bin/kudu
14:03:10.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskhBFXjG/build/release/bin/kudu
14:03:10.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/wal
14:03:10.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/data
14:03:10.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:03:10.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:03:10.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:03:10.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:03:10.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:03:10.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:03:10.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/logs
14:03:10.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/data/info.pb
14:03:10.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:03:10.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:03:10.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:03:10.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:03:10.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:03:10.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:03:10.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:03:10.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:03:10.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:03:10.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:03:10.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.89.66:34685
14:03:10.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.89.66
14:03:10.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.89.126:36189,127.10.89.125:42399,127.10.89.124:42407
14:03:10.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=39225
14:03:10.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.89.66
14:03:10.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.89.84:33487
14:03:10.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:03:10.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:03:10.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:03:10.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:10.913360 12330 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:03:10.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:10.913558 12330 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:03:10.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:10.913590 12330 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:03:10.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:10.915309 12330 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:03:10.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:10.915398 12330 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.89.66
14:03:10.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.917016 12330 tablet_server_runner.cc:78] Tablet server non-default flags:
14:03:10.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/logs
14:03:10.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:03:10.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:03:10.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:03:10.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.89.84:33487
14:03:10.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:03:10.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:03:10.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/data
14:03:10.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/wal
14:03:10.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:03:10.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:03:10.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.89.66:34685
14:03:10.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:03:10.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:03:10.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:03:10.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/data/info.pb
14:03:10.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.89.66
14:03:10.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=39225
14:03:10.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.89.126:36189,127.10.89.125:42399,127.10.89.124:42407
14:03:10.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:03:10.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.12330
14:03:10.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:03:10.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:03:10.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:03:10.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:03:10.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.89.66
14:03:10.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:03:10.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:03:10.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:03:10.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 4bcd631c449d3c6649730ca6e07660c8168f0d44
14:03:10.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
14:03:10.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Aug 2026 13:43:13 UTC on e7f111948823
14:03:10.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 13289
14:03:10.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.917299 12330 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:03:10.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.917573 12330 file_cache.cc:504] Constructed file cache file cache with capacity 419430
14:03:10.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:10.920722 12338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:03:10.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:10.920672 12340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:03:10.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:10.920683 12337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:03:10.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.921043 12330 server_base.cc:1061] running on GCE node
14:03:10.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.921223 12330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:03:10.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.921484 12330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:03:10.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.922638 12330 hybrid_clock.cc:648] HybridClock initialized: now 1786456990922612 us; error 35 us; skew 500 ppm
14:03:10.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.923782 12330 webserver.cc:533] Webserver started at http://127.10.89.66:39225/ using document root <none> and password file <none>
14:03:10.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.924031 12330 fs_manager.cc:362] Metadata directory not provided
14:03:10.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.924104 12330 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:03:10.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.925447 12330 fs_manager.cc:714] Time spent opening directory manager: real 0.001s	user 0.001s	sys 0.000s
14:03:10.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.926218 12347 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:03:10.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.926409 12330 fs_manager.cc:730] Time spent opening block manager: real 0.001s	user 0.000s	sys 0.000s
14:03:10.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.926492 12330 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/data,/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/wal
14:03:10.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "7bf0930626894cff94471c02ef2b9196"
14:03:10.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 14:02:53 on dist-test-slave-g350"
14:03:10.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.926820 12330 fs_report.cc:389] FS layout report
14:03:10.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:03:10.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/wal
14:03:10.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/wal
14:03:10.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/data/data
14:03:10.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:03:10.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:03:10.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:03:10.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:03:10.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:03:10.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:03:10.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:03:10.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:03:10.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:03:10.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:03:10.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:03:10.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.938313 12330 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:03:10.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.938668 12330 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:03:10.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.938822 12330 kserver.cc:163] Server-wide thread pool size limit: 3276
14:03:10.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.939118 12330 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:03:10.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.939666 12354 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
14:03:10.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.940651 12330 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:03:10.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.940724 12330 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s	user 0.000s	sys 0.000s
14:03:10.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.940783 12330 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:03:10.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.941401 12330 ts_tablet_manager.cc:616] Registered 1 tablets
14:03:10.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.941459 12330 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s	user 0.000s	sys 0.000s
14:03:10.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.941485 12354 tablet_bootstrap.cc:492] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196: Bootstrap starting.
14:03:10.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.948691 12330 rpc_server.cc:307] RPC server started. Bound to: 127.10.89.66:34685
14:03:10.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.948762 12479 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.89.66:34685 every 8 connection(s)
14:03:10.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.949129 12330 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/data/info.pb
14:03:10.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.951910 10597 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskhBFXjG/build/release/bin/kudu as pid 12330
14:03:10.952 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:03:10.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.955614 12354 log.cc:826] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196: Log is configured to *not* fsync() on all Append() calls
14:03:10.957 [DEBUG - kudu-nio-7] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 18 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823, tablet=28da754494014b9f85434e4ebbbef9d7, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:03:10.957 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.10.89.65:36825
14:03:10.957 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

14:03:10.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.958102 10597 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskhBFXjG/build/release/bin/kudu with pid 11909
14:03:10.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.959045 12354 tablet_bootstrap.cc:492] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=20 ignored=0} inserts{seen=15 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:03:10.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.959355 12354 tablet_bootstrap.cc:492] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196: Bootstrap complete.
14:03:10.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.960817 12354 ts_tablet_manager.cc:1403] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196: Time spent bootstrapping tablet: real 0.019s	user 0.009s	sys 0.006s
14:03:10.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.962432 12354 raft_consensus.cc:359] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:10.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.962765 12354 raft_consensus.cc:740] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bf0930626894cff94471c02ef2b9196, State: Initialized, Role: FOLLOWER
14:03:10.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.962872 12354 consensus_queue.cc:260] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 20, Last appended: 6.20, Last appended by leader: 20, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:10.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.963131 12354 ts_tablet_manager.cc:1434] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
14:03:10.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.963510 12481 heartbeater.cc:344] Connected to a master server at 127.10.89.126:36189
14:03:10.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.963613 12481 heartbeater.cc:461] Registering TS with master...
14:03:10.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.963689 12482 heartbeater.cc:344] Connected to a master server at 127.10.89.125:42399
14:03:10.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.963742 12482 heartbeater.cc:461] Registering TS with master...
14:03:10.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.963835 12481 heartbeater.cc:507] Master 127.10.89.126:36189 requested a full tablet report, sending...
14:03:10.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.963855 12482 heartbeater.cc:507] Master 127.10.89.125:42399 requested a full tablet report, sending...
14:03:10.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.964166 12480 heartbeater.cc:344] Connected to a master server at 127.10.89.124:42407
14:03:10.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.964223 12480 heartbeater.cc:461] Registering TS with master...
14:03:10.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.964327 12480 heartbeater.cc:507] Master 127.10.89.124:42407 requested a full tablet report, sending...
14:03:10.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.964546 10644 ts_manager.cc:194] Re-registered known tserver with Master: 7bf0930626894cff94471c02ef2b9196 (127.10.89.66:34685)
14:03:10.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.964830 10731 ts_manager.cc:194] Re-registered known tserver with Master: 7bf0930626894cff94471c02ef2b9196 (127.10.89.66:34685)
14:03:10.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.964851 10813 ts_manager.cc:194] Re-registered known tserver with Master: 7bf0930626894cff94471c02ef2b9196 (127.10.89.66:34685)
14:03:10.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.965404 10731 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.10.89.66:54523
14:03:10.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:10.966164 12482 heartbeater.cc:499] Master 127.10.89.125:42399 was elected leader, sending a full tablet report...
14:03:10.968 [DEBUG - kudu-nio-7] (Connection.java:239) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] handling channelInactive
14:03:10.968 [DEBUG - kudu-nio-7] (Connection.java:709) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] cleaning up while in state READY due to: connection closed
14:03:10.969 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:03:10.970 [DEBUG - Time-limited test] (Connection.java:748) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] connecting to peer
14:03:10.971 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:10.971 [DEBUG - kudu-nio-0] (Connection.java:709) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] cleaning up while in state CONNECTING due to: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:10.971 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:10.987 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:10.987 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723),7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685),d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)[L]]}]
14:03:10.987 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=2, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=18), Traces: [0ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [1ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [1ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [14ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [17ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@978852799(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@22b65fa1 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@668f7654 -> wakeup thread Time-limited test)) after lookup
14:03:10.988 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] connecting to peer
14:03:10.990 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:10.990 [DEBUG - kudu-nio-1] (Connection.java:709) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] cleaning up while in state CONNECTING due to: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:10.990 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:11.007 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:11.007 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723),7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685),d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)[L]]}]
14:03:11.007 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=4, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=37), Traces: [0ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [1ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [1ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [14ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [17ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [18ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [20ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [20ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [36ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@978852799(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@22b65fa1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1376574728 after retry RPC@1204643925), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@668f7654 -> wakeup thread Time-limited test -> (continuation of Deferred@1376574728 after retry RPC@1204643925))) after lookup
14:03:11.008 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] connecting to peer
14:03:11.009 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:11.009 [DEBUG - kudu-nio-2] (Connection.java:709) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] cleaning up while in state CONNECTING due to: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:11.009 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:11.028 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:11.028 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723),7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685),d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)[L]]}]
14:03:11.028 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=6, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=58), Traces: [0ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [1ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [1ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [14ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [17ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [18ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [20ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [20ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [36ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [38ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [39ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [40ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [55ms] refreshing cache from master, [55ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [57ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@978852799(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@22b65fa1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1376574728 after retry RPC@1204643925) -> (continuation of Deferred@1763722961 after retry RPC@1166218336), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@668f7654 -> wakeup thread Time-limited test -> (continuation of Deferred@1376574728 after retry RPC@1204643925) -> (continuation of Deferred@1763722961 after retry RPC@1166218336))) after lookup
14:03:11.030 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] connecting to peer
14:03:11.031 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:11.032 [DEBUG - kudu-nio-3] (Connection.java:709) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] cleaning up while in state CONNECTING due to: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:11.032 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:11.067 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:11.067 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723),7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685),d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)[L]]}]
14:03:11.067 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=8, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=97), Traces: [0ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [1ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [1ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [14ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [17ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [18ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [20ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [20ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [36ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [38ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [39ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [40ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [55ms] refreshing cache from master, [55ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [57ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [60ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [62ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [62ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [94ms] refreshing cache from master, [94ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [96ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@978852799(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@22b65fa1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1376574728 after retry RPC@1204643925) -> (continuation of Deferred@1763722961 after retry RPC@1166218336) -> (continuation of Deferred@1919431879 after retry RPC@798297854), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@668f7654 -> wakeup thread Time-limited test -> (continuation of Deferred@1376574728 after retry RPC@1204643925) -> (continuation of Deferred@1763722961 after retry RPC@1166218336) -> (continuation of Deferred@1919431879 after retry RPC@798297854))) after lookup
14:03:11.068 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] connecting to peer
14:03:11.069 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:11.069 [DEBUG - kudu-nio-4] (Connection.java:709) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] cleaning up while in state CONNECTING due to: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:11.069 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:11.467 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:11.467 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723),7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685),d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)[L]]}]
14:03:11.468 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=10, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=498), Traces: [0ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [1ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [1ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [14ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [17ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [18ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [20ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [20ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [36ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [38ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [39ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [40ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [55ms] refreshing cache from master, [55ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [57ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [60ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [62ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [62ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [94ms] refreshing cache from master, [94ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [96ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [98ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [99ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [100ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [494ms] refreshing cache from master, [494ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [497ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@978852799(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@22b65fa1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1376574728 after retry RPC@1204643925) -> (continuation of Deferred@1763722961 after retry RPC@1166218336) -> (continuation of Deferred@1919431879 after retry RPC@798297854) -> (continuation of Deferred@490430050 after retry RPC@450773445), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@668f7654 -> wakeup thread Time-limited test -> (continuation of Deferred@1376574728 after retry RPC@1204643925) -> (continuation of Deferred@1763722961 after retry RPC@1166218336) -> (continuation of Deferred@1919431879 after retry RPC@798297854) -> (continuation of Deferred@490430050 after retry RPC@450773445))) after lookup
14:03:11.470 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] connecting to peer
14:03:11.471 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:11.471 [DEBUG - kudu-nio-5] (Connection.java:709) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] cleaning up while in state CONNECTING due to: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:11.471 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:11.587 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:11.588 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723),7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685),d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)[L]]}]
14:03:11.588 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=12, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=618), Traces: [0ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [1ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [1ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [14ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [17ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [18ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [20ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [20ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [36ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [38ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [39ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [40ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [55ms] refreshing cache from master, [55ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [57ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [60ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [62ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [62ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [94ms] refreshing cache from master, [94ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [96ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [98ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [99ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [100ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [494ms] refreshing cache from master, [494ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [497ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [499ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [501ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [501ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [614ms] refreshing cache from master, [615ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [617ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@978852799(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@22b65fa1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1376574728 after retry RPC@1204643925) -> (continuation of Deferred@1763722961 after retry RPC@1166218336) -> (continuation of Deferred@1919431879 after retry RPC@798297854) -> (continuation of Deferred@490430050 after retry RPC@450773445) -> (continuation of Deferred@225435600 after retry RPC@1700227025), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@668f7654 -> wakeup thread Time-limited test -> (continuation of Deferred@1376574728 after retry RPC@1204643925) -> (continuation of Deferred@1763722961 after retry RPC@1166218336) -> (continuation of Deferred@1919431879 after retry RPC@798297854) -> (continuation of Deferred@490430050 after retry RPC@450773445) -> (continuation of Deferred@225435600 after retry RPC@1700227025))) after lookup
14:03:11.589 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] connecting to peer
14:03:11.590 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:11.590 [DEBUG - kudu-nio-6] (Connection.java:709) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] cleaning up while in state CONNECTING due to: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:11.591 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:12.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:12.323740 12495 raft_consensus.cc:493] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d8b415ec6a9a4671b34e29775c855962)
14:03:12.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:12.323836 12495 raft_consensus.cc:515] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:12.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:12.324033 12495 leader_election.cc:290] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d8b415ec6a9a4671b34e29775c855962 (127.10.89.65:36825), 7bf0930626894cff94471c02ef2b9196 (127.10.89.66:34685)
14:03:12.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:12.324914 12136 leader_election.cc:336] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer d8b415ec6a9a4671b34e29775c855962 (127.10.89.65:36825): Network error: Client connection negotiation failed: client connection to 127.10.89.65:36825: connect: Connection refused (error 111)
14:03:12.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:12.327479 12427 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28da754494014b9f85434e4ebbbef9d7" candidate_uuid: "751f3be0b90f4a1d86af88ca13d42016" candidate_term: 8 candidate_status { last_received { term: 7 index: 24 } } ignore_live_leader: false dest_uuid: "7bf0930626894cff94471c02ef2b9196" is_pre_election: true
14:03:12.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:12.327613 12427 raft_consensus.cc:2468] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 751f3be0b90f4a1d86af88ca13d42016 in term 6.
14:03:12.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:12.327862 12133 leader_election.cc:304] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 751f3be0b90f4a1d86af88ca13d42016, 7bf0930626894cff94471c02ef2b9196; no voters: d8b415ec6a9a4671b34e29775c855962
14:03:12.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:12.327965 12495 raft_consensus.cc:2804] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 7 FOLLOWER]: Leader pre-election won for term 8
14:03:12.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:12.328003 12495 raft_consensus.cc:493] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 7 FOLLOWER]: Starting leader election (detected failure of leader d8b415ec6a9a4671b34e29775c855962)
14:03:12.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:12.328028 12495 raft_consensus.cc:3060] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 7 FOLLOWER]: Advancing to term 8
14:03:12.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:12.328634 12495 raft_consensus.cc:515] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:12.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:12.328846 12495 leader_election.cc:290] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [CANDIDATE]: Term 8 election: Requested vote from peers d8b415ec6a9a4671b34e29775c855962 (127.10.89.65:36825), 7bf0930626894cff94471c02ef2b9196 (127.10.89.66:34685)
14:03:12.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:12.329023 12427 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28da754494014b9f85434e4ebbbef9d7" candidate_uuid: "751f3be0b90f4a1d86af88ca13d42016" candidate_term: 8 candidate_status { last_received { term: 7 index: 24 } } ignore_live_leader: false dest_uuid: "7bf0930626894cff94471c02ef2b9196"
14:03:12.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:12.329113 12427 raft_consensus.cc:3060] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 6 FOLLOWER]: Advancing to term 8
14:03:12.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:12.329455 12136 leader_election.cc:336] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer d8b415ec6a9a4671b34e29775c855962 (127.10.89.65:36825): Network error: Client connection negotiation failed: client connection to 127.10.89.65:36825: connect: Connection refused (error 111)
14:03:12.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:12.330235 12427 raft_consensus.cc:2468] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 751f3be0b90f4a1d86af88ca13d42016 in term 8.
14:03:12.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:12.330425 12133 leader_election.cc:304] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 751f3be0b90f4a1d86af88ca13d42016, 7bf0930626894cff94471c02ef2b9196; no voters: d8b415ec6a9a4671b34e29775c855962
14:03:12.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:12.330528 12495 raft_consensus.cc:2804] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 8 FOLLOWER]: Leader election won for term 8
14:03:12.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:12.330675 12495 raft_consensus.cc:697] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 8 LEADER]: Becoming Leader. State: Replica: 751f3be0b90f4a1d86af88ca13d42016, State: Running, Role: LEADER
14:03:12.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:12.330770 12495 consensus_queue.cc:237] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 7.24, Last appended by leader: 24, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:12.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:12.331540 10731 catalog_manager.cc:5719] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 reported cstate change: term changed from 7 to 8, leader changed from d8b415ec6a9a4671b34e29775c855962 (127.10.89.65) to 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67). New cstate: current_term: 8 leader_uuid: "751f3be0b90f4a1d86af88ca13d42016" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } health_report { overall_health: UNKNOWN } } }
14:03:12.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:12.859025 12136 consensus_peers.cc:597] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 -> Peer d8b415ec6a9a4671b34e29775c855962 (127.10.89.65:36825): Couldn't send request to peer d8b415ec6a9a4671b34e29775c855962. Status: Network error: Client connection negotiation failed: client connection to 127.10.89.65:36825: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:03:12.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:12.953001 12427 raft_consensus.cc:1275] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 8 FOLLOWER]: Refusing update from remote peer 751f3be0b90f4a1d86af88ca13d42016: Log matching property violated. Preceding OpId in replica: term: 6 index: 20. Preceding OpId from leader: term: 8 index: 25. (index mismatch)
14:03:12.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:12.953444 12514 consensus_queue.cc:1048] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 20, Time since last communication: 0.000s
14:03:13.067 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:13.067 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)[L],7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685),d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)]}]
14:03:13.067 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=14, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=2097), Traces: [0ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [1ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [1ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [14ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [17ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [18ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [20ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [20ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [36ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [38ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [39ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [40ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [55ms] refreshing cache from master, [55ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [57ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [60ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [62ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [62ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [94ms] refreshing cache from master, [94ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [96ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [98ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [99ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [100ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [494ms] refreshing cache from master, [494ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [497ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [499ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [501ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [501ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [614ms] refreshing cache from master, [615ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [617ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [619ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [621ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [621ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [2094ms] refreshing cache from master, [2094ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [2096ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@978852799(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@22b65fa1 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1376574728 after retry RPC@1204643925) -> (continuation of Deferred@1763722961 after retry RPC@1166218336) -> (continuation of Deferred@1919431879 after retry RPC@798297854) -> (continuation of Deferred@490430050 after retry RPC@450773445) -> (continuation of Deferred@225435600 after retry RPC@1700227025) -> (continuation of Deferred@875822841 after retry RPC@1778679361), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@668f7654 -> wakeup thread Time-limited test -> (continuation of Deferred@1376574728 after retry RPC@1204643925) -> (continuation of Deferred@1763722961 after retry RPC@1166218336) -> (continuation of Deferred@1919431879 after retry RPC@798297854) -> (continuation of Deferred@490430050 after retry RPC@450773445) -> (continuation of Deferred@225435600 after retry RPC@1700227025) -> (continuation of Deferred@875822841 after retry RPC@1778679361))) after lookup
14:03:13.068 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] connecting to peer
14:03:13.069 [DEBUG - kudu-nio-7] (Connection.java:757) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] Successfully connected to peer
14:03:13.087 [DEBUG - kudu-nio-7] (Negotiator.java:918) Authenticated connection [id: 0x4581e84f, L:/127.0.0.1:33722 - R:/127.10.89.67:46723] using TOKEN/null
14:03:13.099 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.10.89.65:36825
14:03:13.099 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

14:03:13.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.099639 10597 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskhBFXjG/build/release/bin/kudu
14:03:13.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskhBFXjG/build/release/bin/kudu
14:03:13.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/wal
14:03:13.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/data
14:03:13.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:03:13.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:03:13.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:03:13.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:03:13.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:03:13.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:03:13.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/logs
14:03:13.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/data/info.pb
14:03:13.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:03:13.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:03:13.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:03:13.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:03:13.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:03:13.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:03:13.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:03:13.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:03:13.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:03:13.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:03:13.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.89.65:36825
14:03:13.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.89.65
14:03:13.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.89.126:36189,127.10.89.125:42399,127.10.89.124:42407
14:03:13.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43475
14:03:13.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.89.65
14:03:13.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.89.84:33487
14:03:13.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:03:13.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:03:13.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:03:13.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:13.181931 12531 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:03:13.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:13.182098 12531 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:03:13.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:13.182118 12531 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:03:13.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:13.183645 12531 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:03:13.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:13.183696 12531 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.89.65
14:03:13.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.185231 12531 tablet_server_runner.cc:78] Tablet server non-default flags:
14:03:13.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/logs
14:03:13.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:03:13.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:03:13.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:03:13.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.89.84:33487
14:03:13.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:03:13.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:03:13.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/data
14:03:13.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/wal
14:03:13.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:03:13.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:03:13.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.89.65:36825
14:03:13.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:03:13.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:03:13.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:03:13.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/data/info.pb
14:03:13.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.89.65
14:03:13.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43475
14:03:13.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.89.126:36189,127.10.89.125:42399,127.10.89.124:42407
14:03:13.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:03:13.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.12531
14:03:13.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:03:13.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:03:13.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:03:13.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:03:13.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.89.65
14:03:13.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:03:13.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:03:13.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:03:13.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 4bcd631c449d3c6649730ca6e07660c8168f0d44
14:03:13.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
14:03:13.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Aug 2026 13:43:13 UTC on e7f111948823
14:03:13.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 13289
14:03:13.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.185424 12531 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:03:13.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.185657 12531 file_cache.cc:504] Constructed file cache file cache with capacity 419430
14:03:13.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:13.188213 12543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:03:13.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.188402 12531 server_base.cc:1061] running on GCE node
14:03:13.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:13.188248 12540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:03:13.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:13.188340 12541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:03:13.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.188911 12531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:03:13.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.189090 12531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:03:13.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.190258 12531 hybrid_clock.cc:648] HybridClock initialized: now 1786456993190235 us; error 39 us; skew 500 ppm
14:03:13.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.191474 12531 webserver.cc:533] Webserver started at http://127.10.89.65:43475/ using document root <none> and password file <none>
14:03:13.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.191635 12531 fs_manager.cc:362] Metadata directory not provided
14:03:13.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.191672 12531 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:03:13.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.192806 12531 fs_manager.cc:714] Time spent opening directory manager: real 0.001s	user 0.001s	sys 0.000s
14:03:13.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.193409 12551 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:03:13.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.193658 12531 fs_manager.cc:730] Time spent opening block manager: real 0.001s	user 0.000s	sys 0.000s
14:03:13.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.193720 12531 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/data,/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/wal
14:03:13.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d8b415ec6a9a4671b34e29775c855962"
14:03:13.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 14:02:53 on dist-test-slave-g350"
14:03:13.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.193972 12531 fs_report.cc:389] FS layout report
14:03:13.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:03:13.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/wal
14:03:13.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/wal
14:03:13.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/data/data
14:03:13.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:03:13.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:03:13.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:03:13.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:03:13.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:03:13.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:03:13.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:03:13.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:03:13.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:03:13.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:03:13.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:03:13.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.219933 12531 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:03:13.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.220228 12531 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:03:13.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.220315 12531 kserver.cc:163] Server-wide thread pool size limit: 3276
14:03:13.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.220546 12531 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:03:13.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.221072 12558 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
14:03:13.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.222079 12531 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:03:13.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.222127 12531 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s	user 0.000s	sys 0.000s
14:03:13.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.222153 12531 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:03:13.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.222707 12531 ts_tablet_manager.cc:616] Registered 1 tablets
14:03:13.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.222740 12531 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s	user 0.000s	sys 0.000s
14:03:13.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.222842 12558 tablet_bootstrap.cc:492] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962: Bootstrap starting.
14:03:13.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.230458 12531 rpc_server.cc:307] RPC server started. Bound to: 127.10.89.65:36825
14:03:13.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.230561 12688 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.89.65:36825 every 8 connection(s)
14:03:13.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.230908 12531 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/data/info.pb
14:03:13.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.232513 12558 log.cc:826] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962: Log is configured to *not* fsync() on all Append() calls
14:03:13.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.235741 10597 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskhBFXjG/build/release/bin/kudu as pid 12531
14:03:13.236 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:03:13.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.236907 12558 tablet_bootstrap.cc:492] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962: Bootstrap replayed 1/1 log segments. Stats: ops{read=24 overwritten=0 applied=24 ignored=0} inserts{seen=18 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:03:13.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.237221 12558 tablet_bootstrap.cc:492] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962: Bootstrap complete.
14:03:13.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.238474 12558 ts_tablet_manager.cc:1403] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962: Time spent bootstrapping tablet: real 0.016s	user 0.005s	sys 0.009s
14:03:13.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.239494 12558 raft_consensus.cc:359] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:13.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.239782 12558 raft_consensus.cc:740] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b415ec6a9a4671b34e29775c855962, State: Initialized, Role: FOLLOWER
14:03:13.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.239908 12558 consensus_queue.cc:260] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 7.24, Last appended by leader: 24, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:13.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.240146 12691 heartbeater.cc:344] Connected to a master server at 127.10.89.126:36189
14:03:13.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.240149 12558 ts_tablet_manager.cc:1434] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
14:03:13.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.240567 12691 heartbeater.cc:461] Registering TS with master...
14:03:13.241 [DEBUG - kudu-nio-7] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 21 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823, tablet=28da754494014b9f85434e4ebbbef9d7, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:03:13.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.241112 12691 heartbeater.cc:507] Master 127.10.89.126:36189 requested a full tablet report, sending...
14:03:13.242 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.10.89.67:46723
14:03:13.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.241895 10644 ts_manager.cc:194] Re-registered known tserver with Master: d8b415ec6a9a4671b34e29775c855962 (127.10.89.65:36825)
14:03:13.242 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

14:03:13.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.242338 12692 heartbeater.cc:344] Connected to a master server at 127.10.89.125:42399
14:03:13.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.242417 12692 heartbeater.cc:461] Registering TS with master...
14:03:13.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.242514 12692 heartbeater.cc:507] Master 127.10.89.125:42399 requested a full tablet report, sending...
14:03:13.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.242743 10597 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskhBFXjG/build/release/bin/kudu with pid 12116
14:03:13.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.242925 10731 ts_manager.cc:194] Re-registered known tserver with Master: d8b415ec6a9a4671b34e29775c855962 (127.10.89.65:36825)
14:03:13.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.243450 10731 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.10.89.65:46947
14:03:13.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.245318 12689 heartbeater.cc:344] Connected to a master server at 127.10.89.124:42407
14:03:13.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.245388 12689 heartbeater.cc:461] Registering TS with master...
14:03:13.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.245494 12689 heartbeater.cc:507] Master 127.10.89.124:42407 requested a full tablet report, sending...
14:03:13.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.247588 10813 ts_manager.cc:194] Re-registered known tserver with Master: d8b415ec6a9a4671b34e29775c855962 (127.10.89.65:36825)
14:03:13.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:13.247921 12692 heartbeater.cc:499] Master 127.10.89.125:42399 was elected leader, sending a full tablet report...
14:03:13.254 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:03:13.254 [DEBUG - kudu-nio-7] (Connection.java:239) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] handling channelInactive
14:03:13.254 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] cleaning up while in state READY due to: connection closed
14:03:13.265 [DEBUG - pool-2-thread-1] (Connection.java:748) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] connecting to peer
14:03:13.266 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:13.267 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] cleaning up while in state CONNECTING due to: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:13.267 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:13.287 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:13.288 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)[L],7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685),d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)]}]
14:03:13.288 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=3, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=33), Traces: [0ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [1ms] delaying RPC due to: Illegal state: connection is terminated, [10ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [13ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [13ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [33ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@1690820038(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@2d785169 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3749b48b -> wakeup thread Time-limited test)) after lookup
14:03:13.288 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] connecting to peer
14:03:13.289 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:13.290 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] cleaning up while in state CONNECTING due to: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:13.290 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:13.307 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:13.307 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)[L],7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685),d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)]}]
14:03:13.308 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=5, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=53), Traces: [0ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [1ms] delaying RPC due to: Illegal state: connection is terminated, [10ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [13ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [13ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [33ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [34ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [36ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [36ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [50ms] refreshing cache from master, [51ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [53ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@1690820038(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@2d785169 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@215546139 after retry RPC@839870896), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3749b48b -> wakeup thread Time-limited test -> (continuation of Deferred@215546139 after retry RPC@839870896))) after lookup
14:03:13.309 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] connecting to peer
14:03:13.310 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:13.310 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] cleaning up while in state CONNECTING due to: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:13.310 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:13.347 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:13.347 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)[L],7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685),d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)]}]
14:03:13.347 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=7, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=92), Traces: [0ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [1ms] delaying RPC due to: Illegal state: connection is terminated, [10ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [13ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [13ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [33ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [34ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [36ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [36ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [50ms] refreshing cache from master, [51ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [53ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [54ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [56ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [56ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [90ms] refreshing cache from master, [90ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [92ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@1690820038(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@2d785169 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@215546139 after retry RPC@839870896) -> (continuation of Deferred@998656038 after retry RPC@1913517071), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3749b48b -> wakeup thread Time-limited test -> (continuation of Deferred@215546139 after retry RPC@839870896) -> (continuation of Deferred@998656038 after retry RPC@1913517071))) after lookup
14:03:13.347 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] connecting to peer
14:03:13.349 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:13.349 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] cleaning up while in state CONNECTING due to: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:13.349 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:13.447 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:13.448 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)[L],7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685),d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)]}]
14:03:13.448 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=9, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=193), Traces: [0ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [1ms] delaying RPC due to: Illegal state: connection is terminated, [10ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [13ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [13ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [33ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [34ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [36ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [36ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [50ms] refreshing cache from master, [51ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [53ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [54ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [56ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [56ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [90ms] refreshing cache from master, [90ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [92ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [93ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [95ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [95ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [191ms] refreshing cache from master, [191ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [193ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@1690820038(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@2d785169 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@215546139 after retry RPC@839870896) -> (continuation of Deferred@998656038 after retry RPC@1913517071) -> (continuation of Deferred@80063601 after retry RPC@427138410), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3749b48b -> wakeup thread Time-limited test -> (continuation of Deferred@215546139 after retry RPC@839870896) -> (continuation of Deferred@998656038 after retry RPC@1913517071) -> (continuation of Deferred@80063601 after retry RPC@427138410))) after lookup
14:03:13.449 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] connecting to peer
14:03:13.450 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:13.450 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] cleaning up while in state CONNECTING due to: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:13.451 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:14.347 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:14.347 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)[L],7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685),d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)]}]
14:03:14.348 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=11, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=1093), Traces: [0ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [1ms] delaying RPC due to: Illegal state: connection is terminated, [10ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [13ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [13ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [33ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [34ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [36ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [36ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [50ms] refreshing cache from master, [51ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [53ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [54ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [56ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [56ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [90ms] refreshing cache from master, [90ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [92ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [93ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [95ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [95ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [191ms] refreshing cache from master, [191ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [193ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [195ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [196ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [197ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [1091ms] refreshing cache from master, [1091ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [1093ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@1690820038(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@2d785169 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@215546139 after retry RPC@839870896) -> (continuation of Deferred@998656038 after retry RPC@1913517071) -> (continuation of Deferred@80063601 after retry RPC@427138410) -> (continuation of Deferred@328988387 after retry RPC@1770258839), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3749b48b -> wakeup thread Time-limited test -> (continuation of Deferred@215546139 after retry RPC@839870896) -> (continuation of Deferred@998656038 after retry RPC@1913517071) -> (continuation of Deferred@80063601 after retry RPC@427138410) -> (continuation of Deferred@328988387 after retry RPC@1770258839))) after lookup
14:03:14.349 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] connecting to peer
14:03:14.350 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:14.350 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] cleaning up while in state CONNECTING due to: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:14.350 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:14.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:14.599238 12705 raft_consensus.cc:493] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 751f3be0b90f4a1d86af88ca13d42016)
14:03:14.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:14.599375 12705 raft_consensus.cc:515] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:14.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:14.599771 12705 leader_election.cc:290] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723), d8b415ec6a9a4671b34e29775c855962 (127.10.89.65:36825)
14:03:14.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:14.600261 12351 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.89.67:46723: connect: Connection refused (error 111)
14:03:14.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:14.600836 12351 leader_election.cc:336] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723): Network error: Client connection negotiation failed: client connection to 127.10.89.67:46723: connect: Connection refused (error 111)
14:03:14.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:14.603698 12634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28da754494014b9f85434e4ebbbef9d7" candidate_uuid: "7bf0930626894cff94471c02ef2b9196" candidate_term: 9 candidate_status { last_received { term: 8 index: 28 } } ignore_live_leader: false dest_uuid: "d8b415ec6a9a4671b34e29775c855962" is_pre_election: true
14:03:14.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:14.603889 12634 raft_consensus.cc:2468] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bf0930626894cff94471c02ef2b9196 in term 7.
14:03:14.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:14.604233 12351 leader_election.cc:304] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7bf0930626894cff94471c02ef2b9196, d8b415ec6a9a4671b34e29775c855962; no voters: 751f3be0b90f4a1d86af88ca13d42016
14:03:14.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:14.604420 12705 raft_consensus.cc:2804] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 8 FOLLOWER]: Leader pre-election won for term 9
14:03:14.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:14.604555 12705 raft_consensus.cc:493] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 751f3be0b90f4a1d86af88ca13d42016)
14:03:14.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:14.604583 12705 raft_consensus.cc:3060] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 8 FOLLOWER]: Advancing to term 9
14:03:14.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:14.605309 12705 raft_consensus.cc:515] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:14.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:14.605439 12705 leader_election.cc:290] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [CANDIDATE]: Term 9 election: Requested vote from peers 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723), d8b415ec6a9a4671b34e29775c855962 (127.10.89.65:36825)
14:03:14.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:14.605834 12634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28da754494014b9f85434e4ebbbef9d7" candidate_uuid: "7bf0930626894cff94471c02ef2b9196" candidate_term: 9 candidate_status { last_received { term: 8 index: 28 } } ignore_live_leader: false dest_uuid: "d8b415ec6a9a4671b34e29775c855962"
14:03:14.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:14.605952 12634 raft_consensus.cc:3060] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 7 FOLLOWER]: Advancing to term 9
14:03:14.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:14.606148 12351 leader_election.cc:336] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723): Network error: Client connection negotiation failed: client connection to 127.10.89.67:46723: connect: Connection refused (error 111)
14:03:14.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:14.607084 12634 raft_consensus.cc:2468] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bf0930626894cff94471c02ef2b9196 in term 9.
14:03:14.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:14.607304 12351 leader_election.cc:304] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7bf0930626894cff94471c02ef2b9196, d8b415ec6a9a4671b34e29775c855962; no voters: 751f3be0b90f4a1d86af88ca13d42016
14:03:14.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:14.607455 12705 raft_consensus.cc:2804] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 9 FOLLOWER]: Leader election won for term 9
14:03:14.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:14.607604 12705 raft_consensus.cc:697] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 9 LEADER]: Becoming Leader. State: Replica: 7bf0930626894cff94471c02ef2b9196, State: Running, Role: LEADER
14:03:14.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:14.607710 12705 consensus_queue.cc:237] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 8.28, Last appended by leader: 28, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:14.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:14.608573 10731 catalog_manager.cc:5719] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 reported cstate change: term changed from 8 to 9, leader changed from 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67) to 7bf0930626894cff94471c02ef2b9196 (127.10.89.66). New cstate: current_term: 9 leader_uuid: "7bf0930626894cff94471c02ef2b9196" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } health_report { overall_health: HEALTHY } } }
14:03:15.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:15.127286 12634 raft_consensus.cc:1275] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 9 FOLLOWER]: Refusing update from remote peer 7bf0930626894cff94471c02ef2b9196: Log matching property violated. Preceding OpId in replica: term: 7 index: 24. Preceding OpId from leader: term: 9 index: 29. (index mismatch)
14:03:15.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:15.127671 12724 consensus_queue.cc:1048] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 24, Time since last communication: 0.000s
14:03:15.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:15.129647 12726 mvcc.cc:204] Tried to move back new op lower bound from 7317327843724898304 to 7317327840587317248. Current Snapshot: MvccSnapshot[applied={T|T < 7317327843692564480}]
14:03:15.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:15.131289 12351 consensus_peers.cc:597] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 -> Peer 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723): Couldn't send request to peer 751f3be0b90f4a1d86af88ca13d42016. Status: Network error: Client connection negotiation failed: client connection to 127.10.89.67:46723: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:03:17.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:17.602723 12351 consensus_peers.cc:597] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 -> Peer 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723): Couldn't send request to peer 751f3be0b90f4a1d86af88ca13d42016. Status: Network error: Client connection negotiation failed: client connection to 127.10.89.67:46723: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
14:03:18.248 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:18.248 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723),7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)[L],d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)]}]
14:03:18.248 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=13, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=4994), Traces: [0ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [1ms] delaying RPC due to: Illegal state: connection is terminated, [10ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [13ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [13ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [31ms] refreshing cache from master, [31ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [33ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [34ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [36ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [36ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [50ms] refreshing cache from master, [51ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [53ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [54ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [56ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [56ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [90ms] refreshing cache from master, [90ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [92ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [93ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [95ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [95ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [191ms] refreshing cache from master, [191ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [193ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [195ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [196ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [197ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [1091ms] refreshing cache from master, [1091ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [1093ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [1094ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [1096ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [1096ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [4990ms] refreshing cache from master, [4991ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [4993ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@1690820038(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@2d785169 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@215546139 after retry RPC@839870896) -> (continuation of Deferred@998656038 after retry RPC@1913517071) -> (continuation of Deferred@80063601 after retry RPC@427138410) -> (continuation of Deferred@328988387 after retry RPC@1770258839) -> (continuation of Deferred@1248065028 after retry RPC@52616806), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3749b48b -> wakeup thread Time-limited test -> (continuation of Deferred@215546139 after retry RPC@839870896) -> (continuation of Deferred@998656038 after retry RPC@1913517071) -> (continuation of Deferred@80063601 after retry RPC@427138410) -> (continuation of Deferred@328988387 after retry RPC@1770258839) -> (continuation of Deferred@1248065028 after retry RPC@52616806))) after lookup
14:03:18.250 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] connecting to peer
14:03:18.251 [DEBUG - kudu-nio-6] (Connection.java:757) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] Successfully connected to peer
14:03:18.282 [DEBUG - kudu-nio-6] (Negotiator.java:918) Authenticated connection [id: 0x4532b7f5, L:/127.0.0.1:52068 - R:/127.10.89.66:34685] using TOKEN/null
14:03:18.301 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.10.89.67:46723
14:03:18.301 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

14:03:18.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.301795 10597 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskhBFXjG/build/release/bin/kudu
14:03:18.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskhBFXjG/build/release/bin/kudu
14:03:18.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/wal
14:03:18.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/data
14:03:18.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:03:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:03:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:03:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:03:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:03:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:03:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/logs
14:03:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/data/info.pb
14:03:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:03:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:03:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:03:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:03:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:03:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:03:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:03:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:03:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:03:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:03:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.89.67:46723
14:03:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.89.67
14:03:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.89.126:36189,127.10.89.125:42399,127.10.89.124:42407
14:03:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34339
14:03:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.89.67
14:03:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.89.84:33487
14:03:18.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:03:18.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:03:18.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:03:18.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:18.388036 12761 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:03:18.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:18.388236 12761 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:03:18.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:18.388268 12761 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:03:18.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:18.390010 12761 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:03:18.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:18.390091 12761 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.89.67
14:03:18.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.391803 12761 tablet_server_runner.cc:78] Tablet server non-default flags:
14:03:18.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/logs
14:03:18.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:03:18.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:03:18.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:03:18.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.89.84:33487
14:03:18.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:03:18.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:03:18.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/data
14:03:18.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/wal
14:03:18.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:03:18.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:03:18.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.89.67:46723
14:03:18.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:03:18.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:03:18.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:03:18.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/data/info.pb
14:03:18.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.89.67
14:03:18.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34339
14:03:18.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.89.126:36189,127.10.89.125:42399,127.10.89.124:42407
14:03:18.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:03:18.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.12761
14:03:18.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:03:18.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:03:18.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:03:18.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:03:18.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.89.67
14:03:18.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:03:18.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:03:18.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:03:18.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 4bcd631c449d3c6649730ca6e07660c8168f0d44
14:03:18.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
14:03:18.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Aug 2026 13:43:13 UTC on e7f111948823
14:03:18.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 13289
14:03:18.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.392079 12761 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:03:18.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.392362 12761 file_cache.cc:504] Constructed file cache file cache with capacity 419430
14:03:18.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:18.395430 12771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:03:18.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:18.395481 12767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:03:18.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.395510 12761 server_base.cc:1061] running on GCE node
14:03:18.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:18.395430 12769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:03:18.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.396128 12761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:03:18.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.396389 12761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:03:18.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.397619 12761 hybrid_clock.cc:648] HybridClock initialized: now 1786456998397549 us; error 57 us; skew 500 ppm
14:03:18.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.398854 12761 webserver.cc:533] Webserver started at http://127.10.89.67:34339/ using document root <none> and password file <none>
14:03:18.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.399081 12761 fs_manager.cc:362] Metadata directory not provided
14:03:18.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.399159 12761 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:03:18.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.400460 12761 fs_manager.cc:714] Time spent opening directory manager: real 0.001s	user 0.001s	sys 0.000s
14:03:18.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.401242 12777 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:03:18.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.401453 12761 fs_manager.cc:730] Time spent opening block manager: real 0.001s	user 0.001s	sys 0.000s
14:03:18.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.401563 12761 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/data,/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/wal
14:03:18.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "751f3be0b90f4a1d86af88ca13d42016"
14:03:18.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 14:02:53 on dist-test-slave-g350"
14:03:18.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.401916 12761 fs_report.cc:389] FS layout report
14:03:18.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:03:18.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/wal
14:03:18.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/wal
14:03:18.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/data/data
14:03:18.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:03:18.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:03:18.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:03:18.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:03:18.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:03:18.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:03:18.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:03:18.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:03:18.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:03:18.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:03:18.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:03:18.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.432447 12761 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:03:18.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.432845 12761 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:03:18.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.432981 12761 kserver.cc:163] Server-wide thread pool size limit: 3276
14:03:18.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.433259 12761 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:03:18.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.433874 12784 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
14:03:18.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.434842 12761 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:03:18.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.434911 12761 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s	user 0.000s	sys 0.000s
14:03:18.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.434947 12761 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:03:18.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.435559 12761 ts_tablet_manager.cc:616] Registered 1 tablets
14:03:18.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.435616 12761 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s	user 0.000s	sys 0.000s
14:03:18.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.435659 12784 tablet_bootstrap.cc:492] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016: Bootstrap starting.
14:03:18.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.443399 12761 rpc_server.cc:307] RPC server started. Bound to: 127.10.89.67:46723
14:03:18.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.443526 12905 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.89.67:46723 every 8 connection(s)
14:03:18.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.443845 12761 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/data/info.pb
14:03:18.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.447098 12784 log.cc:826] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016: Log is configured to *not* fsync() on all Append() calls
14:03:18.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.448474 10597 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskhBFXjG/build/release/bin/kudu as pid 12761
14:03:18.448 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:03:18.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.451339 12784 tablet_bootstrap.cc:492] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=21 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:03:18.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.451715 12784 tablet_bootstrap.cc:492] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016: Bootstrap complete.
14:03:18.453 [DEBUG - kudu-nio-6] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 24 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823, tablet=28da754494014b9f85434e4ebbbef9d7, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:03:18.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.453104 12784 ts_tablet_manager.cc:1403] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016: Time spent bootstrapping tablet: real 0.017s	user 0.003s	sys 0.010s
14:03:18.455 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.10.89.66:34685
14:03:18.455 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

14:03:18.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.454332 12784 raft_consensus.cc:359] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:18.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.454617 12784 raft_consensus.cc:740] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 751f3be0b90f4a1d86af88ca13d42016, State: Initialized, Role: FOLLOWER
14:03:18.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.454756 12784 consensus_queue.cc:260] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 8.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:18.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.455046 12784 ts_tablet_manager.cc:1434] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.002s
14:03:18.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.455739 10597 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskhBFXjG/build/release/bin/kudu with pid 12330
14:03:18.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.455770 12907 heartbeater.cc:344] Connected to a master server at 127.10.89.126:36189
14:03:18.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.455860 12907 heartbeater.cc:461] Registering TS with master...
14:03:18.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.456089 12907 heartbeater.cc:507] Master 127.10.89.126:36189 requested a full tablet report, sending...
14:03:18.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.457370 10643 ts_manager.cc:194] Re-registered known tserver with Master: 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723)
14:03:18.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.461727 12908 heartbeater.cc:344] Connected to a master server at 127.10.89.125:42399
14:03:18.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.461836 12908 heartbeater.cc:461] Registering TS with master...
14:03:18.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.461961 12908 heartbeater.cc:507] Master 127.10.89.125:42399 requested a full tablet report, sending...
14:03:18.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.462275 12906 heartbeater.cc:344] Connected to a master server at 127.10.89.124:42407
14:03:18.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.462297 10731 ts_manager.cc:194] Re-registered known tserver with Master: 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723)
14:03:18.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.462343 12906 heartbeater.cc:461] Registering TS with master...
14:03:18.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.462441 12906 heartbeater.cc:507] Master 127.10.89.124:42407 requested a full tablet report, sending...
14:03:18.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.462862 10813 ts_manager.cc:194] Re-registered known tserver with Master: 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723)
14:03:18.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.462935 10731 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.10.89.67:42115
14:03:18.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:18.463711 12908 heartbeater.cc:499] Master 127.10.89.125:42399 was elected leader, sending a full tablet report...
14:03:18.468 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:03:18.468 [DEBUG - kudu-nio-6] (Connection.java:239) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] handling channelInactive
14:03:18.468 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] cleaning up while in state READY due to: connection closed
14:03:18.485 [DEBUG - pool-2-thread-1] (Connection.java:748) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] connecting to peer
14:03:18.486 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:18.487 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:18.487 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:18.507 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:18.507 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723),7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)[L],d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)]}]
14:03:18.507 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=3, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=39), Traces: [0ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [0ms] delaying RPC due to: Illegal state: connection is terminated, [17ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [19ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [19ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [38ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@794830392(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@3aa41c5c -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3c33b309 -> wakeup thread Time-limited test)) after lookup
14:03:18.508 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] connecting to peer
14:03:18.509 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:18.509 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:18.509 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:18.527 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:18.527 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723),7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)[L],d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)]}]
14:03:18.528 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=5, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=59), Traces: [0ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [0ms] delaying RPC due to: Illegal state: connection is terminated, [17ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [19ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [19ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [38ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [39ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [41ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [41ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [57ms] refreshing cache from master, [57ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [59ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@794830392(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@3aa41c5c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1129849748 after retry RPC@1657961229), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3c33b309 -> wakeup thread Time-limited test -> (continuation of Deferred@1129849748 after retry RPC@1657961229))) after lookup
14:03:18.529 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] connecting to peer
14:03:18.530 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:18.530 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:18.530 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:18.607 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:18.608 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723),7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)[L],d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)]}]
14:03:18.608 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=7, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=140), Traces: [0ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [0ms] delaying RPC due to: Illegal state: connection is terminated, [17ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [19ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [19ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [38ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [39ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [41ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [41ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [57ms] refreshing cache from master, [57ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [59ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [61ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [62ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [62ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [137ms] refreshing cache from master, [137ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [139ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@794830392(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@3aa41c5c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1129849748 after retry RPC@1657961229) -> (continuation of Deferred@2086210675 after retry RPC@1924064045), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3c33b309 -> wakeup thread Time-limited test -> (continuation of Deferred@1129849748 after retry RPC@1657961229) -> (continuation of Deferred@2086210675 after retry RPC@1924064045))) after lookup
14:03:18.609 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] connecting to peer
14:03:18.610 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:18.611 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:18.611 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:18.867 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:18.867 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723),7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)[L],d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)]}]
14:03:18.868 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=9, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=399), Traces: [0ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [0ms] delaying RPC due to: Illegal state: connection is terminated, [17ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [19ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [19ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [38ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [39ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [41ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [41ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [57ms] refreshing cache from master, [57ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [59ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [61ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [62ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [62ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [137ms] refreshing cache from master, [137ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [139ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [141ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [143ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [143ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [396ms] refreshing cache from master, [397ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [399ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@794830392(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@3aa41c5c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1129849748 after retry RPC@1657961229) -> (continuation of Deferred@2086210675 after retry RPC@1924064045) -> (continuation of Deferred@1871306260 after retry RPC@1790705475), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3c33b309 -> wakeup thread Time-limited test -> (continuation of Deferred@1129849748 after retry RPC@1657961229) -> (continuation of Deferred@2086210675 after retry RPC@1924064045) -> (continuation of Deferred@1871306260 after retry RPC@1790705475))) after lookup
14:03:18.869 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] connecting to peer
14:03:18.870 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:18.870 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:18.870 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:19.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:19.581087 12921 raft_consensus.cc:493] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
14:03:19.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:19.581322 12921 raft_consensus.cc:515] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:19.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:19.581746 12921 leader_election.cc:290] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d8b415ec6a9a4671b34e29775c855962 (127.10.89.65:36825), 7bf0930626894cff94471c02ef2b9196 (127.10.89.66:34685)
14:03:19.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:19.582267 12778 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.89.66:34685: connect: Connection refused (error 111)
14:03:19.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:19.582846 12778 leader_election.cc:336] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 7bf0930626894cff94471c02ef2b9196 (127.10.89.66:34685): Network error: Client connection negotiation failed: client connection to 127.10.89.66:34685: connect: Connection refused (error 111)
14:03:19.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:19.585068 12634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28da754494014b9f85434e4ebbbef9d7" candidate_uuid: "751f3be0b90f4a1d86af88ca13d42016" candidate_term: 9 candidate_status { last_received { term: 8 index: 28 } } ignore_live_leader: false dest_uuid: "d8b415ec6a9a4671b34e29775c855962" is_pre_election: true
14:03:19.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:19.585386 12781 leader_election.cc:304] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 751f3be0b90f4a1d86af88ca13d42016; no voters: 7bf0930626894cff94471c02ef2b9196, d8b415ec6a9a4671b34e29775c855962
14:03:19.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:19.585564 12921 raft_consensus.cc:2749] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
14:03:19.627 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:19.627 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723),7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)[L],d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)]}]
14:03:19.628 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=11, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=1159), Traces: [0ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [0ms] delaying RPC due to: Illegal state: connection is terminated, [17ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [19ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [19ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [38ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [39ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [41ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [41ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [57ms] refreshing cache from master, [57ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [59ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [61ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [62ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [62ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [137ms] refreshing cache from master, [137ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [139ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [141ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [143ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [143ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [396ms] refreshing cache from master, [397ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [399ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [400ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [402ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [402ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [1156ms] refreshing cache from master, [1157ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [1159ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@794830392(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@3aa41c5c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1129849748 after retry RPC@1657961229) -> (continuation of Deferred@2086210675 after retry RPC@1924064045) -> (continuation of Deferred@1871306260 after retry RPC@1790705475) -> (continuation of Deferred@2004617967 after retry RPC@1116250788), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3c33b309 -> wakeup thread Time-limited test -> (continuation of Deferred@1129849748 after retry RPC@1657961229) -> (continuation of Deferred@2086210675 after retry RPC@1924064045) -> (continuation of Deferred@1871306260 after retry RPC@1790705475) -> (continuation of Deferred@2004617967 after retry RPC@1116250788))) after lookup
14:03:19.628 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] connecting to peer
14:03:19.629 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:19.630 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] cleaning up while in state CONNECTING due to: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:19.630 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685
14:03:19.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:19.801136 12926 raft_consensus.cc:493] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 7bf0930626894cff94471c02ef2b9196)
14:03:19.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:19.801286 12926 raft_consensus.cc:515] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:19.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:19.801604 12926 leader_election.cc:290] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723), 7bf0930626894cff94471c02ef2b9196 (127.10.89.66:34685)
14:03:19.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:19.802078 12552 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.89.66:34685: connect: Connection refused (error 111)
14:03:19.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:19.802613 12552 leader_election.cc:336] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 7bf0930626894cff94471c02ef2b9196 (127.10.89.66:34685): Network error: Client connection negotiation failed: client connection to 127.10.89.66:34685: connect: Connection refused (error 111)
14:03:19.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:19.804849 12853 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28da754494014b9f85434e4ebbbef9d7" candidate_uuid: "d8b415ec6a9a4671b34e29775c855962" candidate_term: 10 candidate_status { last_received { term: 9 index: 32 } } ignore_live_leader: false dest_uuid: "751f3be0b90f4a1d86af88ca13d42016" is_pre_election: true
14:03:19.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:19.804968 12853 raft_consensus.cc:2468] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8b415ec6a9a4671b34e29775c855962 in term 8.
14:03:19.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:19.805243 12555 leader_election.cc:304] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 751f3be0b90f4a1d86af88ca13d42016, d8b415ec6a9a4671b34e29775c855962; no voters: 7bf0930626894cff94471c02ef2b9196
14:03:19.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:19.805399 12926 raft_consensus.cc:2804] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 9 FOLLOWER]: Leader pre-election won for term 10
14:03:19.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:19.805485 12926 raft_consensus.cc:493] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 7bf0930626894cff94471c02ef2b9196)
14:03:19.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:19.805509 12926 raft_consensus.cc:3060] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 9 FOLLOWER]: Advancing to term 10
14:03:19.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:19.806401 12926 raft_consensus.cc:515] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:19.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:19.806576 12926 leader_election.cc:290] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [CANDIDATE]: Term 10 election: Requested vote from peers 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723), 7bf0930626894cff94471c02ef2b9196 (127.10.89.66:34685)
14:03:19.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:19.806882 12853 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28da754494014b9f85434e4ebbbef9d7" candidate_uuid: "d8b415ec6a9a4671b34e29775c855962" candidate_term: 10 candidate_status { last_received { term: 9 index: 32 } } ignore_live_leader: false dest_uuid: "751f3be0b90f4a1d86af88ca13d42016"
14:03:19.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:19.806989 12853 raft_consensus.cc:3060] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 8 FOLLOWER]: Advancing to term 10
14:03:19.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:19.807274 12552 leader_election.cc:336] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 7bf0930626894cff94471c02ef2b9196 (127.10.89.66:34685): Network error: Client connection negotiation failed: client connection to 127.10.89.66:34685: connect: Connection refused (error 111)
14:03:19.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:19.808375 12853 raft_consensus.cc:2468] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8b415ec6a9a4671b34e29775c855962 in term 10.
14:03:19.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:19.808579 12555 leader_election.cc:304] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 751f3be0b90f4a1d86af88ca13d42016, d8b415ec6a9a4671b34e29775c855962; no voters: 7bf0930626894cff94471c02ef2b9196
14:03:19.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:19.808686 12926 raft_consensus.cc:2804] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 10 FOLLOWER]: Leader election won for term 10
14:03:19.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:19.808858 12926 raft_consensus.cc:697] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 10 LEADER]: Becoming Leader. State: Replica: d8b415ec6a9a4671b34e29775c855962, State: Running, Role: LEADER
14:03:19.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:19.808962 12926 consensus_queue.cc:237] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 9.32, Last appended by leader: 32, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:19.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:19.809736 10731 catalog_manager.cc:5719] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 reported cstate change: term changed from 9 to 10, leader changed from 7bf0930626894cff94471c02ef2b9196 (127.10.89.66) to d8b415ec6a9a4671b34e29775c855962 (127.10.89.65). New cstate: current_term: 10 leader_uuid: "d8b415ec6a9a4671b34e29775c855962" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } health_report { overall_health: UNKNOWN } } }
14:03:20.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:20.243762 12853 raft_consensus.cc:1275] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 10 FOLLOWER]: Refusing update from remote peer d8b415ec6a9a4671b34e29775c855962: Log matching property violated. Preceding OpId in replica: term: 8 index: 28. Preceding OpId from leader: term: 10 index: 33. (index mismatch)
14:03:20.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:20.244246 12926 consensus_queue.cc:1048] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [LEADER]: Connected to new peer: Peer: permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 28, Time since last communication: 0.000s
14:03:20.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:20.247740 12552 consensus_peers.cc:597] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 -> Peer 7bf0930626894cff94471c02ef2b9196 (127.10.89.66:34685): Couldn't send request to peer 7bf0930626894cff94471c02ef2b9196. Status: Network error: Client connection negotiation failed: client connection to 127.10.89.66:34685: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:03:22.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:22.737576 12552 consensus_peers.cc:597] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 -> Peer 7bf0930626894cff94471c02ef2b9196 (127.10.89.66:34685): Couldn't send request to peer 7bf0930626894cff94471c02ef2b9196. Status: Network error: Client connection negotiation failed: client connection to 127.10.89.66:34685: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
14:03:23.167 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:23.168 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723),7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685),d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)[L]]}]
14:03:23.168 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=13, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=4699), Traces: [0ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [0ms] delaying RPC due to: Illegal state: connection is terminated, [17ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [19ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [19ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [38ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [39ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [41ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [41ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [57ms] refreshing cache from master, [57ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [59ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [61ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [62ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [62ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [137ms] refreshing cache from master, [137ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [139ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [141ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [143ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [143ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [396ms] refreshing cache from master, [397ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [399ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [400ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [402ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [402ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [1156ms] refreshing cache from master, [1157ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [1159ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [1160ms] sending RPC to server 7bf0930626894cff94471c02ef2b9196, [1162ms] received response from server 7bf0930626894cff94471c02ef2b9196: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [1162ms] delaying RPC due to: Network error: Failed to connect to peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685): Connection refused: /127.10.89.66:34685, [4696ms] refreshing cache from master, [4697ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [4699ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@794830392(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@3aa41c5c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1129849748 after retry RPC@1657961229) -> (continuation of Deferred@2086210675 after retry RPC@1924064045) -> (continuation of Deferred@1871306260 after retry RPC@1790705475) -> (continuation of Deferred@2004617967 after retry RPC@1116250788) -> (continuation of Deferred@1749221664 after retry RPC@1868108558), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3c33b309 -> wakeup thread Time-limited test -> (continuation of Deferred@1129849748 after retry RPC@1657961229) -> (continuation of Deferred@2086210675 after retry RPC@1924064045) -> (continuation of Deferred@1871306260 after retry RPC@1790705475) -> (continuation of Deferred@2004617967 after retry RPC@1116250788) -> (continuation of Deferred@1749221664 after retry RPC@1868108558))) after lookup
14:03:23.169 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] connecting to peer
14:03:23.170 [DEBUG - kudu-nio-5] (Connection.java:757) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] Successfully connected to peer
14:03:23.199 [DEBUG - kudu-nio-5] (Negotiator.java:918) Authenticated connection [id: 0x7bda4ad0, L:/127.0.0.1:35770 - R:/127.10.89.65:36825] using TOKEN/null
14:03:23.213 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.10.89.66:34685
14:03:23.213 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

14:03:23.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.214334 10597 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskhBFXjG/build/release/bin/kudu
14:03:23.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskhBFXjG/build/release/bin/kudu
14:03:23.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/wal
14:03:23.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/data
14:03:23.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:03:23.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:03:23.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:03:23.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:03:23.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:03:23.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:03:23.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/logs
14:03:23.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/data/info.pb
14:03:23.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:03:23.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:03:23.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:03:23.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:03:23.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:03:23.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:03:23.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:03:23.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:03:23.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:03:23.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:03:23.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.89.66:34685
14:03:23.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.89.66
14:03:23.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.89.126:36189,127.10.89.125:42399,127.10.89.124:42407
14:03:23.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=39225
14:03:23.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.89.66
14:03:23.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.89.84:33487
14:03:23.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:03:23.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:03:23.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:03:23.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:23.305296 12972 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:03:23.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:23.305487 12972 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:03:23.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:23.305558 12972 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:03:23.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:23.307319 12972 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:03:23.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:23.307389 12972 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.89.66
14:03:23.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.309020 12972 tablet_server_runner.cc:78] Tablet server non-default flags:
14:03:23.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/logs
14:03:23.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:03:23.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:03:23.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:03:23.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.89.84:33487
14:03:23.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:03:23.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:03:23.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/data
14:03:23.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/wal
14:03:23.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:03:23.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:03:23.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.89.66:34685
14:03:23.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:03:23.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:03:23.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:03:23.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/data/info.pb
14:03:23.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.89.66
14:03:23.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=39225
14:03:23.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.89.126:36189,127.10.89.125:42399,127.10.89.124:42407
14:03:23.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:03:23.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.12972
14:03:23.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:03:23.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:03:23.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:03:23.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:03:23.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.89.66
14:03:23.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:03:23.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:03:23.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:03:23.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 4bcd631c449d3c6649730ca6e07660c8168f0d44
14:03:23.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
14:03:23.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Aug 2026 13:43:13 UTC on e7f111948823
14:03:23.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 13289
14:03:23.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.309314 12972 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:03:23.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.309631 12972 file_cache.cc:504] Constructed file cache file cache with capacity 419430
14:03:23.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:23.312630 12982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:03:23.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:23.312664 12984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:03:23.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:23.312687 12981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:03:23.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.313158 12972 server_base.cc:1061] running on GCE node
14:03:23.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.313409 12972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:03:23.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.313691 12972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:03:23.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.314872 12972 hybrid_clock.cc:648] HybridClock initialized: now 1786457003314844 us; error 38 us; skew 500 ppm
14:03:23.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.316216 12972 webserver.cc:533] Webserver started at http://127.10.89.66:39225/ using document root <none> and password file <none>
14:03:23.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.316466 12972 fs_manager.cc:362] Metadata directory not provided
14:03:23.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.316545 12972 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:03:23.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.317982 12972 fs_manager.cc:714] Time spent opening directory manager: real 0.001s	user 0.001s	sys 0.000s
14:03:23.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.318768 12992 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:03:23.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.318970 12972 fs_manager.cc:730] Time spent opening block manager: real 0.001s	user 0.001s	sys 0.000s
14:03:23.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.319051 12972 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/data,/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/wal
14:03:23.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "7bf0930626894cff94471c02ef2b9196"
14:03:23.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 14:02:53 on dist-test-slave-g350"
14:03:23.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.319420 12972 fs_report.cc:389] FS layout report
14:03:23.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:03:23.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/wal
14:03:23.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/wal
14:03:23.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/data/data
14:03:23.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:03:23.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:03:23.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:03:23.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:03:23.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:03:23.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:03:23.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:03:23.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:03:23.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:03:23.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:03:23.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:03:23.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.331451 12972 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:03:23.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.331853 12972 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:03:23.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.332031 12972 kserver.cc:163] Server-wide thread pool size limit: 3276
14:03:23.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.332306 12972 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:03:23.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.332901 12999 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
14:03:23.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.333935 12972 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:03:23.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.334002 12972 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s	user 0.000s	sys 0.000s
14:03:23.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.334043 12972 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:03:23.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.334683 12972 ts_tablet_manager.cc:616] Registered 1 tablets
14:03:23.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.334740 12972 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s	user 0.000s	sys 0.000s
14:03:23.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.334787 12999 tablet_bootstrap.cc:492] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196: Bootstrap starting.
14:03:23.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.341933 12972 rpc_server.cc:307] RPC server started. Bound to: 127.10.89.66:34685
14:03:23.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.341981 13119 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.89.66:34685 every 8 connection(s)
14:03:23.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.342332 12972 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-1/data/info.pb
14:03:23.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.344012 10597 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskhBFXjG/build/release/bin/kudu as pid 12972
14:03:23.344 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:03:23.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.346743 12999 log.cc:826] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196: Log is configured to *not* fsync() on all Append() calls
14:03:23.348 [DEBUG - kudu-nio-5] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 27 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823, tablet=28da754494014b9f85434e4ebbbef9d7, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:03:23.349 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.10.89.65:36825
14:03:23.350 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

14:03:23.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.350370 10597 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskhBFXjG/build/release/bin/kudu with pid 12531
14:03:23.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.356187 12999 tablet_bootstrap.cc:492] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=24 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:03:23.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.356642 12999 tablet_bootstrap.cc:492] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196: Bootstrap complete.
14:03:23.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.357178 13120 heartbeater.cc:344] Connected to a master server at 127.10.89.124:42407
14:03:23.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.357293 13120 heartbeater.cc:461] Registering TS with master...
14:03:23.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.357524 13120 heartbeater.cc:507] Master 127.10.89.124:42407 requested a full tablet report, sending...
14:03:23.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.358199 12999 ts_tablet_manager.cc:1403] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196: Time spent bootstrapping tablet: real 0.023s	user 0.007s	sys 0.011s
14:03:23.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.358256 10813 ts_manager.cc:194] Re-registered known tserver with Master: 7bf0930626894cff94471c02ef2b9196 (127.10.89.66:34685)
14:03:23.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.359697 12999 raft_consensus.cc:359] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 9 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:23.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.360033 12999 raft_consensus.cc:740] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bf0930626894cff94471c02ef2b9196, State: Initialized, Role: FOLLOWER
14:03:23.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.360082 13121 heartbeater.cc:344] Connected to a master server at 127.10.89.126:36189
14:03:23.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.360128 13121 heartbeater.cc:461] Registering TS with master...
14:03:23.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.360168 12999 consensus_queue.cc:260] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 9.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:23.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.360234 13121 heartbeater.cc:507] Master 127.10.89.126:36189 requested a full tablet report, sending...
14:03:23.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.360512 12999 ts_tablet_manager.cc:1434] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.002s
14:03:23.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.360616 10643 ts_manager.cc:194] Re-registered known tserver with Master: 7bf0930626894cff94471c02ef2b9196 (127.10.89.66:34685)
14:03:23.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.360644 13123 heartbeater.cc:344] Connected to a master server at 127.10.89.125:42399
14:03:23.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.360687 13123 heartbeater.cc:461] Registering TS with master...
14:03:23.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.360793 13123 heartbeater.cc:507] Master 127.10.89.125:42399 requested a full tablet report, sending...
14:03:23.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.361135 10731 ts_manager.cc:194] Re-registered known tserver with Master: 7bf0930626894cff94471c02ef2b9196 (127.10.89.66:34685)
14:03:23.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.361677 10731 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.10.89.66:58891
14:03:23.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:23.362301 13123 heartbeater.cc:499] Master 127.10.89.125:42399 was elected leader, sending a full tablet report...
14:03:23.366 [DEBUG - kudu-nio-5] (Connection.java:239) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] handling channelInactive
14:03:23.366 [DEBUG - kudu-nio-5] (Connection.java:709) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] cleaning up while in state READY due to: connection closed
14:03:23.367 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:03:23.367 [DEBUG - Time-limited test] (Connection.java:748) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] connecting to peer
14:03:23.368 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:23.369 [DEBUG - kudu-nio-6] (Connection.java:709) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] cleaning up while in state CONNECTING due to: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:23.369 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:23.388 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:23.388 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723),7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685),d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)[L]]}]
14:03:23.388 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=2, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=21), Traces: [0ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [2ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [2ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [18ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [20ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@759057008(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@ad8edba -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@76ffc46e -> wakeup thread Time-limited test)) after lookup
14:03:23.389 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] connecting to peer
14:03:23.390 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:23.390 [DEBUG - kudu-nio-7] (Connection.java:709) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] cleaning up while in state CONNECTING due to: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:23.390 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:23.407 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:23.407 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723),7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685),d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)[L]]}]
14:03:23.407 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=4, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=40), Traces: [0ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [2ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [2ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [18ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [20ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [22ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [23ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [23ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [40ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@759057008(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@ad8edba -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@421211913 after retry RPC@290407664), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@76ffc46e -> wakeup thread Time-limited test -> (continuation of Deferred@421211913 after retry RPC@290407664))) after lookup
14:03:23.408 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] connecting to peer
14:03:23.409 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:23.409 [DEBUG - kudu-nio-0] (Connection.java:709) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] cleaning up while in state CONNECTING due to: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:23.409 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:23.427 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:23.427 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723),7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685),d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)[L]]}]
14:03:23.427 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=6, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=60), Traces: [0ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [2ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [2ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [18ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [20ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [22ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [23ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [23ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [40ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [41ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [42ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [42ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [57ms] refreshing cache from master, [58ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [59ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@759057008(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@ad8edba -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@421211913 after retry RPC@290407664) -> (continuation of Deferred@455240185 after retry RPC@1889650345), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@76ffc46e -> wakeup thread Time-limited test -> (continuation of Deferred@421211913 after retry RPC@290407664) -> (continuation of Deferred@455240185 after retry RPC@1889650345))) after lookup
14:03:23.428 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] connecting to peer
14:03:23.429 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:23.429 [DEBUG - kudu-nio-1] (Connection.java:709) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] cleaning up while in state CONNECTING due to: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:23.429 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:23.447 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:23.447 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723),7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685),d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)[L]]}]
14:03:23.447 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=8, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=80), Traces: [0ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [2ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [2ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [18ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [20ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [22ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [23ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [23ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [40ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [41ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [42ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [42ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [57ms] refreshing cache from master, [58ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [59ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [61ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [62ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [62ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [79ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@759057008(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@ad8edba -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@421211913 after retry RPC@290407664) -> (continuation of Deferred@455240185 after retry RPC@1889650345) -> (continuation of Deferred@1946706598 after retry RPC@1026285720), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@76ffc46e -> wakeup thread Time-limited test -> (continuation of Deferred@421211913 after retry RPC@290407664) -> (continuation of Deferred@455240185 after retry RPC@1889650345) -> (continuation of Deferred@1946706598 after retry RPC@1026285720))) after lookup
14:03:23.448 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] connecting to peer
14:03:23.449 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:23.449 [DEBUG - kudu-nio-2] (Connection.java:709) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] cleaning up while in state CONNECTING due to: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:23.449 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:23.667 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:23.667 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723),7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685),d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)[L]]}]
14:03:23.667 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=10, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=300), Traces: [0ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [2ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [2ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [18ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [20ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [22ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [23ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [23ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [40ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [41ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [42ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [42ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [57ms] refreshing cache from master, [58ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [59ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [61ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [62ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [62ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [79ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [80ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [82ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [82ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [297ms] refreshing cache from master, [298ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [300ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@759057008(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@ad8edba -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@421211913 after retry RPC@290407664) -> (continuation of Deferred@455240185 after retry RPC@1889650345) -> (continuation of Deferred@1946706598 after retry RPC@1026285720) -> (continuation of Deferred@615802268 after retry RPC@1190751922), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@76ffc46e -> wakeup thread Time-limited test -> (continuation of Deferred@421211913 after retry RPC@290407664) -> (continuation of Deferred@455240185 after retry RPC@1889650345) -> (continuation of Deferred@1946706598 after retry RPC@1026285720) -> (continuation of Deferred@615802268 after retry RPC@1190751922))) after lookup
14:03:23.668 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] connecting to peer
14:03:23.669 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:23.669 [DEBUG - kudu-nio-3] (Connection.java:709) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] cleaning up while in state CONNECTING due to: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:23.669 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825
14:03:24.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:24.591584 13137 raft_consensus.cc:493] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
14:03:24.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:24.591722 13137 raft_consensus.cc:515] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:24.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:24.591997 13137 leader_election.cc:290] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723), d8b415ec6a9a4671b34e29775c855962 (127.10.89.65:36825)
14:03:24.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:24.592855 12996 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.89.65:36825: connect: Connection refused (error 111)
14:03:24.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:24.593499 12996 leader_election.cc:336] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer d8b415ec6a9a4671b34e29775c855962 (127.10.89.65:36825): Network error: Client connection negotiation failed: client connection to 127.10.89.65:36825: connect: Connection refused (error 111)
14:03:24.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:24.595497 12853 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28da754494014b9f85434e4ebbbef9d7" candidate_uuid: "7bf0930626894cff94471c02ef2b9196" candidate_term: 10 candidate_status { last_received { term: 9 index: 32 } } ignore_live_leader: false dest_uuid: "751f3be0b90f4a1d86af88ca13d42016" is_pre_election: true
14:03:24.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:24.595860 12996 leader_election.cc:304] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7bf0930626894cff94471c02ef2b9196; no voters: 751f3be0b90f4a1d86af88ca13d42016, d8b415ec6a9a4671b34e29775c855962
14:03:24.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:24.596071 13137 raft_consensus.cc:2749] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
14:03:24.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:24.713374 13143 raft_consensus.cc:493] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d8b415ec6a9a4671b34e29775c855962)
14:03:24.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:24.713518 13143 raft_consensus.cc:515] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:24.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:24.713740 13143 leader_election.cc:290] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d8b415ec6a9a4671b34e29775c855962 (127.10.89.65:36825), 7bf0930626894cff94471c02ef2b9196 (127.10.89.66:34685)
14:03:24.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:24.714186 12781 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.89.65:36825: connect: Connection refused (error 111)
14:03:24.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:24.714651 12781 leader_election.cc:336] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer d8b415ec6a9a4671b34e29775c855962 (127.10.89.65:36825): Network error: Client connection negotiation failed: client connection to 127.10.89.65:36825: connect: Connection refused (error 111)
14:03:24.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:24.717044 13070 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28da754494014b9f85434e4ebbbef9d7" candidate_uuid: "751f3be0b90f4a1d86af88ca13d42016" candidate_term: 11 candidate_status { last_received { term: 10 index: 36 } } ignore_live_leader: false dest_uuid: "7bf0930626894cff94471c02ef2b9196" is_pre_election: true
14:03:24.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:24.717173 13070 raft_consensus.cc:2468] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 751f3be0b90f4a1d86af88ca13d42016 in term 9.
14:03:24.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:24.717432 12778 leader_election.cc:304] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 751f3be0b90f4a1d86af88ca13d42016, 7bf0930626894cff94471c02ef2b9196; no voters: d8b415ec6a9a4671b34e29775c855962
14:03:24.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:24.717568 13143 raft_consensus.cc:2804] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 10 FOLLOWER]: Leader pre-election won for term 11
14:03:24.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:24.717666 13143 raft_consensus.cc:493] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 10 FOLLOWER]: Starting leader election (detected failure of leader d8b415ec6a9a4671b34e29775c855962)
14:03:24.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:24.717684 13143 raft_consensus.cc:3060] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 10 FOLLOWER]: Advancing to term 11
14:03:24.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:24.718354 13143 raft_consensus.cc:515] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:24.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:24.718523 13143 leader_election.cc:290] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [CANDIDATE]: Term 11 election: Requested vote from peers d8b415ec6a9a4671b34e29775c855962 (127.10.89.65:36825), 7bf0930626894cff94471c02ef2b9196 (127.10.89.66:34685)
14:03:24.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:24.718750 13070 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28da754494014b9f85434e4ebbbef9d7" candidate_uuid: "751f3be0b90f4a1d86af88ca13d42016" candidate_term: 11 candidate_status { last_received { term: 10 index: 36 } } ignore_live_leader: false dest_uuid: "7bf0930626894cff94471c02ef2b9196"
14:03:24.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:24.718835 13070 raft_consensus.cc:3060] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 9 FOLLOWER]: Advancing to term 11
14:03:24.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:24.719102 12781 leader_election.cc:336] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer d8b415ec6a9a4671b34e29775c855962 (127.10.89.65:36825): Network error: Client connection negotiation failed: client connection to 127.10.89.65:36825: connect: Connection refused (error 111)
14:03:24.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:24.719905 13070 raft_consensus.cc:2468] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 751f3be0b90f4a1d86af88ca13d42016 in term 11.
14:03:24.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:24.720112 12778 leader_election.cc:304] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 751f3be0b90f4a1d86af88ca13d42016, 7bf0930626894cff94471c02ef2b9196; no voters: d8b415ec6a9a4671b34e29775c855962
14:03:24.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:24.720230 13143 raft_consensus.cc:2804] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 11 FOLLOWER]: Leader election won for term 11
14:03:24.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:24.720376 13143 raft_consensus.cc:697] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 11 LEADER]: Becoming Leader. State: Replica: 751f3be0b90f4a1d86af88ca13d42016, State: Running, Role: LEADER
14:03:24.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:24.720476 13143 consensus_queue.cc:237] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 10.36, Last appended by leader: 36, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:24.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:24.721300 10731 catalog_manager.cc:5719] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 reported cstate change: term changed from 10 to 11, leader changed from d8b415ec6a9a4671b34e29775c855962 (127.10.89.65) to 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67). New cstate: current_term: 11 leader_uuid: "751f3be0b90f4a1d86af88ca13d42016" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } health_report { overall_health: UNKNOWN } } }
14:03:25.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.157038 13070 raft_consensus.cc:1275] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 11 FOLLOWER]: Refusing update from remote peer 751f3be0b90f4a1d86af88ca13d42016: Log matching property violated. Preceding OpId in replica: term: 9 index: 32. Preceding OpId from leader: term: 11 index: 37. (index mismatch)
14:03:25.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.157558 13143 consensus_queue.cc:1048] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 32, Time since last communication: 0.000s
14:03:25.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:25.161216 12781 consensus_peers.cc:597] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 -> Peer d8b415ec6a9a4671b34e29775c855962 (127.10.89.65:36825): Couldn't send request to peer d8b415ec6a9a4671b34e29775c855962. Status: Network error: Client connection negotiation failed: client connection to 127.10.89.65:36825: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:03:25.567 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:25.567 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)[L],7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685),d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)]}]
14:03:25.568 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=12, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=2200), Traces: [0ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [2ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [2ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [18ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [20ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [22ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [23ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [23ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [40ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [41ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [42ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [42ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [57ms] refreshing cache from master, [58ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [59ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [61ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [62ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [62ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [79ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [80ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [82ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [82ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [297ms] refreshing cache from master, [298ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [300ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [301ms] sending RPC to server d8b415ec6a9a4671b34e29775c855962, [302ms] received response from server d8b415ec6a9a4671b34e29775c855962: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [302ms] delaying RPC due to: Network error: Failed to connect to peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825): Connection refused: /127.10.89.65:36825, [2197ms] refreshing cache from master, [2197ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [2199ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@759057008(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@ad8edba -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@421211913 after retry RPC@290407664) -> (continuation of Deferred@455240185 after retry RPC@1889650345) -> (continuation of Deferred@1946706598 after retry RPC@1026285720) -> (continuation of Deferred@615802268 after retry RPC@1190751922) -> (continuation of Deferred@2074993541 after retry RPC@1991024316), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@76ffc46e -> wakeup thread Time-limited test -> (continuation of Deferred@421211913 after retry RPC@290407664) -> (continuation of Deferred@455240185 after retry RPC@1889650345) -> (continuation of Deferred@1946706598 after retry RPC@1026285720) -> (continuation of Deferred@615802268 after retry RPC@1190751922) -> (continuation of Deferred@2074993541 after retry RPC@1991024316))) after lookup
14:03:25.568 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] connecting to peer
14:03:25.569 [DEBUG - kudu-nio-4] (Connection.java:757) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] Successfully connected to peer
14:03:25.597 [DEBUG - kudu-nio-4] (Negotiator.java:918) Authenticated connection [id: 0x622b0314, L:/127.0.0.1:39544 - R:/127.10.89.67:46723] using TOKEN/null
14:03:25.614 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.10.89.65:36825
14:03:25.614 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

14:03:25.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.614877 10597 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskhBFXjG/build/release/bin/kudu
14:03:25.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskhBFXjG/build/release/bin/kudu
14:03:25.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/wal
14:03:25.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/data
14:03:25.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:03:25.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:03:25.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:03:25.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:03:25.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:03:25.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:03:25.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/logs
14:03:25.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/data/info.pb
14:03:25.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:03:25.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:03:25.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:03:25.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:03:25.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:03:25.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:03:25.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:03:25.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:03:25.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:03:25.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:03:25.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.89.65:36825
14:03:25.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.89.65
14:03:25.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.89.126:36189,127.10.89.125:42399,127.10.89.124:42407
14:03:25.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43475
14:03:25.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.89.65
14:03:25.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.89.84:33487
14:03:25.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:03:25.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:03:25.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:03:25.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:25.706107 13174 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:03:25.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:25.706316 13174 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:03:25.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:25.706341 13174 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:03:25.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:25.708019 13174 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:03:25.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:25.708093 13174 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.89.65
14:03:25.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.709802 13174 tablet_server_runner.cc:78] Tablet server non-default flags:
14:03:25.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/logs
14:03:25.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:03:25.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:03:25.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:03:25.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.89.84:33487
14:03:25.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:03:25.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:03:25.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/data
14:03:25.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/wal
14:03:25.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:03:25.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:03:25.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.89.65:36825
14:03:25.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:03:25.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:03:25.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:03:25.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/data/info.pb
14:03:25.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.89.65
14:03:25.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=43475
14:03:25.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.89.126:36189,127.10.89.125:42399,127.10.89.124:42407
14:03:25.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:03:25.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.13174
14:03:25.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:03:25.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:03:25.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:03:25.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:03:25.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.89.65
14:03:25.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:03:25.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:03:25.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:03:25.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 4bcd631c449d3c6649730ca6e07660c8168f0d44
14:03:25.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
14:03:25.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Aug 2026 13:43:13 UTC on e7f111948823
14:03:25.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 13289
14:03:25.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.710017 13174 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:03:25.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.710239 13174 file_cache.cc:504] Constructed file cache file cache with capacity 419430
14:03:25.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:25.713177 13180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:03:25.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.713188 13174 server_base.cc:1061] running on GCE node
14:03:25.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:25.713200 13182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:03:25.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:25.713197 13179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:03:25.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.713675 13174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:03:25.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.713960 13174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:03:25.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.715137 13174 hybrid_clock.cc:648] HybridClock initialized: now 1786457005715111 us; error 43 us; skew 500 ppm
14:03:25.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.716507 13174 webserver.cc:533] Webserver started at http://127.10.89.65:43475/ using document root <none> and password file <none>
14:03:25.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.716769 13174 fs_manager.cc:362] Metadata directory not provided
14:03:25.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.716871 13174 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:03:25.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.718271 13174 fs_manager.cc:714] Time spent opening directory manager: real 0.001s	user 0.002s	sys 0.000s
14:03:25.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.719010 13190 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:03:25.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.719197 13174 fs_manager.cc:730] Time spent opening block manager: real 0.001s	user 0.001s	sys 0.000s
14:03:25.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.719300 13174 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/data,/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/wal
14:03:25.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d8b415ec6a9a4671b34e29775c855962"
14:03:25.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 14:02:53 on dist-test-slave-g350"
14:03:25.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.719617 13174 fs_report.cc:389] FS layout report
14:03:25.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:03:25.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/wal
14:03:25.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/wal
14:03:25.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/data/data
14:03:25.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:03:25.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:03:25.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:03:25.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:03:25.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:03:25.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:03:25.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:03:25.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:03:25.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:03:25.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:03:25.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:03:25.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.730216 13174 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:03:25.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.730588 13174 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:03:25.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.730763 13174 kserver.cc:163] Server-wide thread pool size limit: 3276
14:03:25.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.731015 13174 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:03:25.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.731518 13197 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
14:03:25.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.732332 13174 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:03:25.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.732421 13174 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s	user 0.000s	sys 0.000s
14:03:25.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.732463 13174 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:03:25.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.733091 13174 ts_tablet_manager.cc:616] Registered 1 tablets
14:03:25.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.733196 13174 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s	user 0.000s	sys 0.000s
14:03:25.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.733203 13197 tablet_bootstrap.cc:492] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962: Bootstrap starting.
14:03:25.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.741204 13174 rpc_server.cc:307] RPC server started. Bound to: 127.10.89.65:36825
14:03:25.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.741246 13324 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.89.65:36825 every 8 connection(s)
14:03:25.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.741688 13174 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-0/data/info.pb
14:03:25.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.743952 13197 log.cc:826] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962: Log is configured to *not* fsync() on all Append() calls
14:03:25.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.749159 13197 tablet_bootstrap.cc:492] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962: Bootstrap replayed 1/1 log segments. Stats: ops{read=36 overwritten=0 applied=36 ignored=0} inserts{seen=27 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:03:25.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.749440 13197 tablet_bootstrap.cc:492] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962: Bootstrap complete.
14:03:25.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.751087 10597 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskhBFXjG/build/release/bin/kudu as pid 13174
14:03:25.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.751351 13326 heartbeater.cc:344] Connected to a master server at 127.10.89.126:36189
14:03:25.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.750860 13197 ts_tablet_manager.cc:1403] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962: Time spent bootstrapping tablet: real 0.018s	user 0.002s	sys 0.013s
14:03:25.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.751454 13326 heartbeater.cc:461] Registering TS with master...
14:03:25.751 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:03:25.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.751668 13326 heartbeater.cc:507] Master 127.10.89.126:36189 requested a full tablet report, sending...
14:03:25.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.752154 13325 heartbeater.cc:344] Connected to a master server at 127.10.89.124:42407
14:03:25.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.752213 13325 heartbeater.cc:461] Registering TS with master...
14:03:25.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.752305 13325 heartbeater.cc:507] Master 127.10.89.124:42407 requested a full tablet report, sending...
14:03:25.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.752349 10643 ts_manager.cc:194] Re-registered known tserver with Master: d8b415ec6a9a4671b34e29775c855962 (127.10.89.65:36825)
14:03:25.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.752612 10814 ts_manager.cc:194] Re-registered known tserver with Master: d8b415ec6a9a4671b34e29775c855962 (127.10.89.65:36825)
14:03:25.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.752727 13197 raft_consensus.cc:359] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 10 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:25.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.753021 13197 raft_consensus.cc:740] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 10 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b415ec6a9a4671b34e29775c855962, State: Initialized, Role: FOLLOWER
14:03:25.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.753129 13197 consensus_queue.cc:260] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 10.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:25.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.753392 13197 ts_tablet_manager.cc:1434] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
14:03:25.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.753506 13327 heartbeater.cc:344] Connected to a master server at 127.10.89.125:42399
14:03:25.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.753568 13327 heartbeater.cc:461] Registering TS with master...
14:03:25.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.753666 13327 heartbeater.cc:507] Master 127.10.89.125:42399 requested a full tablet report, sending...
14:03:25.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.753947 10731 ts_manager.cc:194] Re-registered known tserver with Master: d8b415ec6a9a4671b34e29775c855962 (127.10.89.65:36825)
14:03:25.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.754477 10731 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.10.89.65:43513
14:03:25.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.755297 13327 heartbeater.cc:499] Master 127.10.89.125:42399 was elected leader, sending a full tablet report...
14:03:25.755 [DEBUG - kudu-nio-4] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 30 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823, tablet=28da754494014b9f85434e4ebbbef9d7, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:03:25.756 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.10.89.67:46723
14:03:25.756 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

14:03:25.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:25.756567 10597 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskhBFXjG/build/release/bin/kudu with pid 12761
14:03:25.768 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:03:25.768 [DEBUG - kudu-nio-4] (Connection.java:239) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] handling channelInactive
14:03:25.768 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] cleaning up while in state READY due to: connection closed
14:03:25.785 [DEBUG - pool-2-thread-1] (Connection.java:748) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] connecting to peer
14:03:25.787 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:25.787 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] cleaning up while in state CONNECTING due to: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:25.787 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:25.807 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:25.807 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)[L],7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685),d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)]}]
14:03:25.807 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=3, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=39), Traces: [0ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [1ms] delaying RPC due to: Illegal state: connection is terminated, [17ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [19ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [19ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [38ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@1700780810(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@7794f063 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@30c4769b -> wakeup thread Time-limited test)) after lookup
14:03:25.808 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] connecting to peer
14:03:25.809 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:25.809 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] cleaning up while in state CONNECTING due to: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:25.809 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:25.826 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:25.827 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)[L],7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685),d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)]}]
14:03:25.827 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=5, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=59), Traces: [0ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [1ms] delaying RPC due to: Illegal state: connection is terminated, [17ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [19ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [19ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [38ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [39ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [41ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [41ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [56ms] refreshing cache from master, [56ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [58ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@1700780810(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@7794f063 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1606055654 after retry RPC@1280946980), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@30c4769b -> wakeup thread Time-limited test -> (continuation of Deferred@1606055654 after retry RPC@1280946980))) after lookup
14:03:25.828 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] connecting to peer
14:03:25.829 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:25.829 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] cleaning up while in state CONNECTING due to: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:25.829 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:25.887 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:25.888 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)[L],7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685),d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)]}]
14:03:25.888 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=7, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=120), Traces: [0ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [1ms] delaying RPC due to: Illegal state: connection is terminated, [17ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [19ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [19ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [38ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [39ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [41ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [41ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [56ms] refreshing cache from master, [56ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [58ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [59ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [61ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [61ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [117ms] refreshing cache from master, [117ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [119ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@1700780810(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@7794f063 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1606055654 after retry RPC@1280946980) -> (continuation of Deferred@995063711 after retry RPC@974005057), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@30c4769b -> wakeup thread Time-limited test -> (continuation of Deferred@1606055654 after retry RPC@1280946980) -> (continuation of Deferred@995063711 after retry RPC@974005057))) after lookup
14:03:25.889 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] connecting to peer
14:03:25.891 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:25.891 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] cleaning up while in state CONNECTING due to: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:25.891 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:25.928 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:25.928 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)[L],7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685),d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)]}]
14:03:25.929 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=9, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=160), Traces: [0ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [1ms] delaying RPC due to: Illegal state: connection is terminated, [17ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [19ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [19ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [38ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [39ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [41ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [41ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [56ms] refreshing cache from master, [56ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [58ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [59ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [61ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [61ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [117ms] refreshing cache from master, [117ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [119ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [120ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [123ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [123ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [157ms] refreshing cache from master, [157ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [160ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@1700780810(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@7794f063 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1606055654 after retry RPC@1280946980) -> (continuation of Deferred@995063711 after retry RPC@974005057) -> (continuation of Deferred@904273730 after retry RPC@344138927), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@30c4769b -> wakeup thread Time-limited test -> (continuation of Deferred@1606055654 after retry RPC@1280946980) -> (continuation of Deferred@995063711 after retry RPC@974005057) -> (continuation of Deferred@904273730 after retry RPC@344138927))) after lookup
14:03:25.930 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] connecting to peer
14:03:25.931 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:25.931 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] cleaning up while in state CONNECTING due to: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:25.931 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:26.787 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:26.787 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)[L],7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685),d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)]}]
14:03:26.788 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=11, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=1019), Traces: [0ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [1ms] delaying RPC due to: Illegal state: connection is terminated, [17ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [19ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [19ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [38ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [39ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [41ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [41ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [56ms] refreshing cache from master, [56ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [58ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [59ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [61ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [61ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [117ms] refreshing cache from master, [117ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [119ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [120ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [123ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [123ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [157ms] refreshing cache from master, [157ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [160ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [161ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [163ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [163ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [1017ms] refreshing cache from master, [1017ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [1019ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@1700780810(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@7794f063 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1606055654 after retry RPC@1280946980) -> (continuation of Deferred@995063711 after retry RPC@974005057) -> (continuation of Deferred@904273730 after retry RPC@344138927) -> (continuation of Deferred@507959961 after retry RPC@1260876751), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@30c4769b -> wakeup thread Time-limited test -> (continuation of Deferred@1606055654 after retry RPC@1280946980) -> (continuation of Deferred@995063711 after retry RPC@974005057) -> (continuation of Deferred@904273730 after retry RPC@344138927) -> (continuation of Deferred@507959961 after retry RPC@1260876751))) after lookup
14:03:26.788 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] connecting to peer
14:03:26.789 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:26.789 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] cleaning up while in state CONNECTING due to: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:26.790 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723) for tablet 28da754494014b9f85434e4ebbbef9d7: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723
14:03:27.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:27.111193 13340 raft_consensus.cc:493] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 751f3be0b90f4a1d86af88ca13d42016)
14:03:27.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:27.111316 13340 raft_consensus.cc:515] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:27.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:27.111513 13340 leader_election.cc:290] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723), d8b415ec6a9a4671b34e29775c855962 (127.10.89.65:36825)
14:03:27.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:27.112519 12996 leader_election.cc:336] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723): Network error: Client connection negotiation failed: client connection to 127.10.89.67:46723: connect: Connection refused (error 111)
14:03:27.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:27.115361 13273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28da754494014b9f85434e4ebbbef9d7" candidate_uuid: "7bf0930626894cff94471c02ef2b9196" candidate_term: 12 candidate_status { last_received { term: 11 index: 40 } } ignore_live_leader: false dest_uuid: "d8b415ec6a9a4671b34e29775c855962" is_pre_election: true
14:03:27.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:27.115545 13273 raft_consensus.cc:2468] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bf0930626894cff94471c02ef2b9196 in term 10.
14:03:27.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:27.115842 12996 leader_election.cc:304] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7bf0930626894cff94471c02ef2b9196, d8b415ec6a9a4671b34e29775c855962; no voters: 751f3be0b90f4a1d86af88ca13d42016
14:03:27.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:27.116010 13340 raft_consensus.cc:2804] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 11 FOLLOWER]: Leader pre-election won for term 12
14:03:27.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:27.116070 13340 raft_consensus.cc:493] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 751f3be0b90f4a1d86af88ca13d42016)
14:03:27.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:27.116086 13340 raft_consensus.cc:3060] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 11 FOLLOWER]: Advancing to term 12
14:03:27.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:27.116708 13340 raft_consensus.cc:515] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:27.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:27.116843 13340 leader_election.cc:290] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [CANDIDATE]: Term 12 election: Requested vote from peers 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723), d8b415ec6a9a4671b34e29775c855962 (127.10.89.65:36825)
14:03:27.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:27.117296 13273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28da754494014b9f85434e4ebbbef9d7" candidate_uuid: "7bf0930626894cff94471c02ef2b9196" candidate_term: 12 candidate_status { last_received { term: 11 index: 40 } } ignore_live_leader: false dest_uuid: "d8b415ec6a9a4671b34e29775c855962"
14:03:27.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:27.117403 13273 raft_consensus.cc:3060] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 10 FOLLOWER]: Advancing to term 12
14:03:27.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:27.117574 12996 leader_election.cc:336] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723): Network error: Client connection negotiation failed: client connection to 127.10.89.67:46723: connect: Connection refused (error 111)
14:03:27.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:27.118713 13273 raft_consensus.cc:2468] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bf0930626894cff94471c02ef2b9196 in term 12.
14:03:27.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:27.119004 12996 leader_election.cc:304] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7bf0930626894cff94471c02ef2b9196, d8b415ec6a9a4671b34e29775c855962; no voters: 751f3be0b90f4a1d86af88ca13d42016
14:03:27.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:27.119158 13340 raft_consensus.cc:2804] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 12 FOLLOWER]: Leader election won for term 12
14:03:27.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:27.119375 13340 raft_consensus.cc:697] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [term 12 LEADER]: Becoming Leader. State: Replica: 7bf0930626894cff94471c02ef2b9196, State: Running, Role: LEADER
14:03:27.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:27.119499 13340 consensus_queue.cc:237] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 11.40, Last appended by leader: 40, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:27.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:27.120366 10731 catalog_manager.cc:5719] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 reported cstate change: term changed from 11 to 12, leader changed from 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67) to 7bf0930626894cff94471c02ef2b9196 (127.10.89.66). New cstate: current_term: 12 leader_uuid: "7bf0930626894cff94471c02ef2b9196" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } health_report { overall_health: HEALTHY } } }
14:03:27.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:27.562525 13273 raft_consensus.cc:1275] T 28da754494014b9f85434e4ebbbef9d7 P d8b415ec6a9a4671b34e29775c855962 [term 12 FOLLOWER]: Refusing update from remote peer 7bf0930626894cff94471c02ef2b9196: Log matching property violated. Preceding OpId in replica: term: 10 index: 36. Preceding OpId from leader: term: 12 index: 41. (index mismatch)
14:03:27.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:27.562966 13340 consensus_queue.cc:1048] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 36, Time since last communication: 0.000s
14:03:27.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:27.566896 12996 consensus_peers.cc:597] T 28da754494014b9f85434e4ebbbef9d7 P 7bf0930626894cff94471c02ef2b9196 -> Peer 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723): Couldn't send request to peer 751f3be0b90f4a1d86af88ca13d42016. Status: Network error: Client connection negotiation failed: client connection to 127.10.89.67:46723: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:03:27.887 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 28da754494014b9f85434e4ebbbef9d7 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823' with partition [<start>, <end>)
14:03:27.887 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=28da754494014b9f85434e4ebbbef9d7@[751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723),7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)[L],d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)]}]
14:03:27.887 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=28da754494014b9f85434e4ebbbef9d7, attempt=13, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=2119), Traces: [0ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [1ms] delaying RPC due to: Illegal state: connection is terminated, [17ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [19ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [19ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [38ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [39ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [41ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [41ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [56ms] refreshing cache from master, [56ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [58ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [59ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [61ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [61ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [117ms] refreshing cache from master, [117ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [119ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [120ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [123ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [123ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [157ms] refreshing cache from master, [157ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [160ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [161ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [163ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [163ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [1017ms] refreshing cache from master, [1017ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [1019ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, [1020ms] sending RPC to server 751f3be0b90f4a1d86af88ca13d42016, [1021ms] received response from server 751f3be0b90f4a1d86af88ca13d42016: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [1022ms] delaying RPC due to: Network error: Failed to connect to peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723): Connection refused: /127.10.89.67:46723, [2117ms] refreshing cache from master, [2117ms] Sub RPC GetTableLocations: sending RPC to server master-127.10.89.125:42399, [2119ms] Sub RPC GetTableLocations: received response from server master-127.10.89.125:42399: OK, deferred=Deferred@1700780810(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$240/0x00007f55c83391e8@7794f063 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1606055654 after retry RPC@1280946980) -> (continuation of Deferred@995063711 after retry RPC@974005057) -> (continuation of Deferred@904273730 after retry RPC@344138927) -> (continuation of Deferred@507959961 after retry RPC@1260876751) -> (continuation of Deferred@828600248 after retry RPC@1093323600), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@30c4769b -> wakeup thread Time-limited test -> (continuation of Deferred@1606055654 after retry RPC@1280946980) -> (continuation of Deferred@995063711 after retry RPC@974005057) -> (continuation of Deferred@904273730 after retry RPC@344138927) -> (continuation of Deferred@507959961 after retry RPC@1260876751) -> (continuation of Deferred@828600248 after retry RPC@1093323600))) after lookup
14:03:27.888 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] connecting to peer
14:03:27.889 [DEBUG - kudu-nio-3] (Connection.java:757) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] Successfully connected to peer
14:03:27.903 [DEBUG - kudu-nio-3] (Negotiator.java:918) Authenticated connection [id: 0x856c48d5, L:/127.0.0.1:52130 - R:/127.10.89.66:34685] using TOKEN/null
14:03:27.913 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.10.89.67:46723
14:03:27.913 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

14:03:27.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:27.914103 10597 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskhBFXjG/build/release/bin/kudu
14:03:27.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskhBFXjG/build/release/bin/kudu
14:03:27.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/wal
14:03:27.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/data
14:03:27.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:03:27.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:03:27.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:03:27.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:03:27.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:03:27.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:03:27.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/logs
14:03:27.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/data/info.pb
14:03:27.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:03:27.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:03:27.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:03:27.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:03:27.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:03:27.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:03:27.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:03:27.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:03:27.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:03:27.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:03:27.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.89.67:46723
14:03:27.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.89.67
14:03:27.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.89.126:36189,127.10.89.125:42399,127.10.89.124:42407
14:03:27.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34339
14:03:27.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.89.67
14:03:27.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.89.84:33487
14:03:27.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:03:27.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:03:27.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:03:28.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:28.003011 13369 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:03:28.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:28.003183 13369 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:03:28.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:28.003203 13369 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:03:28.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:28.004675 13369 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:03:28.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:28.004725 13369 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.89.67
14:03:28.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.006474 13369 tablet_server_runner.cc:78] Tablet server non-default flags:
14:03:28.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/logs
14:03:28.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:03:28.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:03:28.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:03:28.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.10.89.84:33487
14:03:28.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:03:28.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:03:28.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/data
14:03:28.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/wal
14:03:28.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:03:28.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:03:28.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.10.89.67:46723
14:03:28.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:03:28.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:03:28.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:03:28.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/data/info.pb
14:03:28.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.10.89.67
14:03:28.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34339
14:03:28.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.10.89.126:36189,127.10.89.125:42399,127.10.89.124:42407
14:03:28.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:03:28.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.13369
14:03:28.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:03:28.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:03:28.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:03:28.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:03:28.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.10.89.67
14:03:28.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:03:28.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:03:28.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:03:28.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 4bcd631c449d3c6649730ca6e07660c8168f0d44
14:03:28.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type RELEASE
14:03:28.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 11 Aug 2026 13:43:13 UTC on e7f111948823
14:03:28.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 13289
14:03:28.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.006677 13369 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:03:28.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.006861 13369 file_cache.cc:504] Constructed file cache file cache with capacity 419430
14:03:28.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:28.009835 13376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:03:28.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:28.009847 13377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:03:28.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.010176 13369 server_base.cc:1061] running on GCE node
14:03:28.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260811 14:03:28.009841 13379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:03:28.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.011581 13369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:03:28.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.011794 13369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:03:28.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.012975 13369 hybrid_clock.cc:648] HybridClock initialized: now 1786457008012943 us; error 48 us; skew 500 ppm
14:03:28.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.014403 13369 webserver.cc:533] Webserver started at http://127.10.89.67:34339/ using document root <none> and password file <none>
14:03:28.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.014668 13369 fs_manager.cc:362] Metadata directory not provided
14:03:28.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.014776 13369 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:03:28.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.016249 13369 fs_manager.cc:714] Time spent opening directory manager: real 0.001s	user 0.000s	sys 0.001s
14:03:28.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.017055 13386 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:03:28.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.017382 13369 fs_manager.cc:730] Time spent opening block manager: real 0.001s	user 0.000s	sys 0.001s
14:03:28.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.017490 13369 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/data,/tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/wal
14:03:28.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "751f3be0b90f4a1d86af88ca13d42016"
14:03:28.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-11 14:02:53 on dist-test-slave-g350"
14:03:28.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.017875 13369 fs_report.cc:389] FS layout report
14:03:28.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:03:28.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/wal
14:03:28.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/wal
14:03:28.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/data/data
14:03:28.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:03:28.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:03:28.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:03:28.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:03:28.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:03:28.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:03:28.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:03:28.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:03:28.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:03:28.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:03:28.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:03:28.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.028883 13369 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:03:28.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.029363 13369 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:03:28.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.029526 13369 kserver.cc:163] Server-wide thread pool size limit: 3276
14:03:28.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.029808 13369 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:03:28.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.030367 13396 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
14:03:28.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.031284 13369 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:03:28.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.031360 13369 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s	user 0.000s	sys 0.000s
14:03:28.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.031401 13369 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:03:28.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.032012 13369 ts_tablet_manager.cc:616] Registered 1 tablets
14:03:28.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.032070 13369 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s	user 0.000s	sys 0.000s
14:03:28.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.032105 13396 tablet_bootstrap.cc:492] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016: Bootstrap starting.
14:03:28.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.042606 13369 rpc_server.cc:307] RPC server started. Bound to: 127.10.89.67:46723
14:03:28.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.042640 13523 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.10.89.67:46723 every 8 connection(s)
14:03:28.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.043165 13369 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskhBFXjG/test-tmp/mini-kudu-cluster13805628794902850675/ts-2/data/info.pb
14:03:28.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.047164 13396 log.cc:826] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016: Log is configured to *not* fsync() on all Append() calls
14:03:28.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.049543 13525 heartbeater.cc:344] Connected to a master server at 127.10.89.126:36189
14:03:28.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.049551 13526 heartbeater.cc:344] Connected to a master server at 127.10.89.125:42399
14:03:28.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.049710 13526 heartbeater.cc:461] Registering TS with master...
14:03:28.050 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:03:28.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.049708 13525 heartbeater.cc:461] Registering TS with master...
14:03:28.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.049887 13525 heartbeater.cc:507] Master 127.10.89.126:36189 requested a full tablet report, sending...
14:03:28.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.050405 10597 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskhBFXjG/build/release/bin/kudu as pid 13369
14:03:28.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.051137 13524 heartbeater.cc:344] Connected to a master server at 127.10.89.124:42407
14:03:28.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.051200 13524 heartbeater.cc:461] Registering TS with master...
14:03:28.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.051306 13524 heartbeater.cc:507] Master 127.10.89.124:42407 requested a full tablet report, sending...
14:03:28.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.051538 10643 ts_manager.cc:194] Re-registered known tserver with Master: 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723)
14:03:28.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.051971 10814 ts_manager.cc:194] Re-registered known tserver with Master: 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723)
14:03:28.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.053318 13396 tablet_bootstrap.cc:492] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016: Bootstrap replayed 1/1 log segments. Stats: ops{read=40 overwritten=0 applied=40 ignored=0} inserts{seen=30 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:03:28.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.053700 13396 tablet_bootstrap.cc:492] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016: Bootstrap complete.
14:03:28.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.055088 13396 ts_tablet_manager.cc:1403] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016: Time spent bootstrapping tablet: real 0.023s	user 0.002s	sys 0.017s
14:03:28.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.056236 13396 raft_consensus.cc:359] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 11 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:28.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.056501 13396 raft_consensus.cc:740] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [term 11 FOLLOWER]: Becoming Follower/Learner. State: Replica: 751f3be0b90f4a1d86af88ca13d42016, State: Initialized, Role: FOLLOWER
14:03:28.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.056613 13396 consensus_queue.cc:260] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 11.40, Last appended by leader: 40, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "751f3be0b90f4a1d86af88ca13d42016" member_type: VOTER last_known_addr { host: "127.10.89.67" port: 46723 } } peers { permanent_uuid: "d8b415ec6a9a4671b34e29775c855962" member_type: VOTER last_known_addr { host: "127.10.89.65" port: 36825 } } peers { permanent_uuid: "7bf0930626894cff94471c02ef2b9196" member_type: VOTER last_known_addr { host: "127.10.89.66" port: 34685 } }
14:03:28.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.056895 13396 ts_tablet_manager.cc:1434] T 28da754494014b9f85434e4ebbbef9d7 P 751f3be0b90f4a1d86af88ca13d42016: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.003s
14:03:28.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.049891 13526 heartbeater.cc:507] Master 127.10.89.125:42399 requested a full tablet report, sending...
14:03:28.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.057600 10731 ts_manager.cc:194] Re-registered known tserver with Master: 751f3be0b90f4a1d86af88ca13d42016 (127.10.89.67:46723)
14:03:28.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.058135 10731 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.10.89.67:33017
14:03:28.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.058840 13526 heartbeater.cc:499] Master 127.10.89.125:42399 was elected leader, sending a full tablet report...
14:03:28.059 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823, tablet=28da754494014b9f85434e4ebbbef9d7, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:03:28.061 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1786456973823, tablet=28da754494014b9f85434e4ebbbef9d7, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:03:28.066 [DEBUG - main] (Connection.java:575) [peer d8b415ec6a9a4671b34e29775c855962(127.10.89.65:36825)] disconnecting while in state TERMINATED
14:03:28.067 [DEBUG - main] (Connection.java:575) [peer master-127.10.89.126:36189(127.10.89.126:36189)] disconnecting while in state READY
14:03:28.067 [DEBUG - main] (Connection.java:575) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] disconnecting while in state READY
14:03:28.067 [DEBUG - main] (Connection.java:575) [peer master-127.10.89.125:42399(127.10.89.125:42399)] disconnecting while in state READY
14:03:28.067 [DEBUG - main] (Connection.java:575) [peer master-127.10.89.124:42407(127.10.89.124:42407)] disconnecting while in state READY
14:03:28.067 [DEBUG - main] (Connection.java:575) [peer 751f3be0b90f4a1d86af88ca13d42016(127.10.89.67:46723)] disconnecting while in state TERMINATED
14:03:28.070 [DEBUG - kudu-nio-2] (Connection.java:239) [peer master-127.10.89.126:36189(127.10.89.126:36189)] handling channelInactive
14:03:28.070 [DEBUG - kudu-nio-2] (Connection.java:709) [peer master-127.10.89.126:36189(127.10.89.126:36189)] cleaning up while in state READY due to: connection closed
14:03:28.070 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.10.89.124:42407(127.10.89.124:42407)] handling channelInactive
14:03:28.070 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.10.89.125:42399(127.10.89.125:42399)] handling channelInactive
14:03:28.070 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.10.89.124:42407(127.10.89.124:42407)] cleaning up while in state READY due to: connection closed
14:03:28.070 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.10.89.125:42399(127.10.89.125:42399)] cleaning up while in state READY due to: connection closed
14:03:28.072 [DEBUG - kudu-nio-3] (AsyncKuduClient.java:2704) Releasing all remaining resources
14:03:28.072 [DEBUG - kudu-nio-3] (Connection.java:239) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] handling channelInactive
14:03:28.073 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 7bf0930626894cff94471c02ef2b9196(127.10.89.66:34685)] cleaning up while in state READY due to: connection closed
14:03:28.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.074378 10597 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskhBFXjG/build/release/bin/kudu with pid 13174
14:03:28.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.092922 10597 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskhBFXjG/build/release/bin/kudu with pid 12972
14:03:28.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.102700 10597 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskhBFXjG/build/release/bin/kudu with pid 13369
14:03:28.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.110730 10597 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskhBFXjG/build/release/bin/kudu with pid 10610
14:03:28.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.120690 10597 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskhBFXjG/build/release/bin/kudu with pid 10696
14:03:28.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260811 14:03:28.129066 10597 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskhBFXjG/build/release/bin/kudu with pid 10773
14:03:28.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-08-11T14:03:28Z chronyd exiting

Time: 36.93

OK (1 test)

