JUnit version 4.13.2
.02:03:35.007 [DEBUG - main] (RandomUtils.java:49) Using random seed: 1779069814999
02:03:35.055 [INFO - main] (RetryRule.java:138) Creating RetryStatement with result reporter and retry count of 0 (no retries)
02:03:35.108 [INFO - main] (KuduTestHarness.java:155) Creating a new MiniKuduCluster...
02:03:35.119 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskxPorqF/test-tmp
02:03:35.124 [INFO - main] (KuduBinaryLocator.java:83) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/debug/bin
02:03:35.126 [INFO - main] (MiniKuduCluster.java:217) Starting process: [/tmp/dist-test-taskxPorqF/java/build/dist-test/../../../build/debug/bin/kudu, test, mini_cluster, --serialization=pb]
02:03:35.308 [DEBUG - main] (MiniKuduCluster.java:185) Request: create_cluster {
  num_masters: 3
  num_tservers: 3
  enable_kerberos: false
  cluster_root: "/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732"
  hms_mode: NONE
  mini_kdc_options {
  }
  principal: "kudu"
  mini_oidc_options {
  }
}

02:03:35.320 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
02:03:35.322 [DEBUG - main] (MiniKuduCluster.java:185) Request: start_cluster {
}

02:03:35.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.324409 30358 mini_oidc.cc:150] Starting JWKS server
02:03:35.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.326681 30358 webserver.cc:533] Webserver started at http://127.0.0.1:43275/ using document root <none> and password file <none>
02:03:35.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.326925 30358 mini_oidc.cc:186] Starting OIDC Discovery server
02:03:35.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.327190 30358 webserver.cc:533] Webserver started at http://127.0.0.1:35381/ using document root <none> and password file <none>
02:03:35.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-05-18T02:03:35Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
02:03:35.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-05-18T02:03:35Z Disabled control of system clock
02:03:35.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.344264 30358 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskxPorqF/build/debug/bin/kudu
02:03:35.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskxPorqF/build/debug/bin/kudu
02:03:35.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-0/wal
02:03:35.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-0/data
02:03:35.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:03:35.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:03:35.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:03:35.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:03:35.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:03:35.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:03:35.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-0/logs
02:03:35.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-0/data/info.pb
02:03:35.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:03:35.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:03:35.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:03:35.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:03:35.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:03:35.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:03:35.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:03:35.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:03:35.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
02:03:35.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:03:35.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
02:03:35.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
02:03:35.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.165.190:39217
02:03:35.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.165.190
02:03:35.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:03:35.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.165.148:38003
02:03:35.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:03:35.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:03:35.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:03:35.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
02:03:35.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.29.165.190:39217,127.29.165.189:39733,127.29.165.188:38583
02:03:35.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
02:03:35.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:43275/jwks.json with env {}
02:03:35.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.447095 30369 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:03:35.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.447294 30369 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:03:35.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.447337 30369 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:03:35.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.451052 30369 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
02:03:35.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.451115 30369 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:03:35.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.451139 30369 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
02:03:35.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.451159 30369 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
02:03:35.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.451179 30369 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
02:03:35.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.451196 30369 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:43275/jwks.json
02:03:35.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.455319 30369 master_runner.cc:387] Master server non-default flags:
02:03:35.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:03:35.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.165.148:38003
02:03:35.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:03:35.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:03:35.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-0/data
02:03:35.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-0/wal
02:03:35.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
02:03:35.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.29.165.190:39217,127.29.165.189:39733,127.29.165.188:38583
02:03:35.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:03:35.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:03:35.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
02:03:35.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.165.190:39217
02:03:35.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
02:03:35.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:03:35.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
02:03:35.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:43275/jwks.json
02:03:35.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:03:35.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:03:35.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-0/data/info.pb
02:03:35.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.165.190
02:03:35.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:03:35.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:03:35.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.30369
02:03:35.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:03:35.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:03:35.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:03:35.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:03:35.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-0/logs
02:03:35.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:03:35.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:03:35.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:03:35.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
02:03:35.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:03:35.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
02:03:35.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type DEBUG
02:03:35.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 01:43:15 UTC on bdcb31816ec0
02:03:35.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11948
02:03:35.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.456472 30369 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:03:35.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.457793 30369 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:03:35.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.464143 30377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:35.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.464161 30374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:35.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.464178 30375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:35.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.464696 30369 server_base.cc:1061] running on GCE node
02:03:35.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.465067 30369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:03:35.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.465863 30369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:03:35.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.467026 30369 hybrid_clock.cc:648] HybridClock initialized: now 1779069815467008 us; error 38 us; skew 500 ppm
02:03:35.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.468667 30369 webserver.cc:533] Webserver started at http://127.29.165.190:35987/ using document root <none> and password file <none>
02:03:35.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.469275 30369 fs_manager.cc:362] Metadata directory not provided
02:03:35.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.469352 30369 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:03:35.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.469563 30369 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
02:03:35.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.471159 30369 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-0/data/instance:
02:03:35.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "324afe9044bf40299572cad6ecfc4d5d"
02:03:35.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:03:35 on dist-test-slave-838s"
02:03:35.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.471662 30369 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-0/wal/instance:
02:03:35.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "324afe9044bf40299572cad6ecfc4d5d"
02:03:35.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:03:35 on dist-test-slave-838s"
02:03:35.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.475083 30369 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.003s	sys 0.001s
02:03:35.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.477087 30383 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:03:35.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.477988 30369 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
02:03:35.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.478103 30369 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-0/data,/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-0/wal
02:03:35.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "324afe9044bf40299572cad6ecfc4d5d"
02:03:35.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:03:35 on dist-test-slave-838s"
02:03:35.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.478188 30369 fs_report.cc:389] FS layout report
02:03:35.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:03:35.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-0/wal
02:03:35.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-0/wal
02:03:35.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-0/data/data
02:03:35.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:03:35.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:03:35.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:03:35.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:03:35.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:03:35.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:03:35.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:03:35.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:03:35.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:03:35.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:03:35.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:03:35.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.489557 30369 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:03:35.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.490199 30369 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:03:35.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.490379 30369 kserver.cc:163] Server-wide thread pool size limit: 3276
02:03:35.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.496891 30369 rpc_server.cc:307] RPC server started. Bound to: 127.29.165.190:39217
02:03:35.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.496901 30435 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.165.190:39217 every 8 connection(s)
02:03:35.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.497926 30369 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-0/data/info.pb
02:03:35.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.500641 30358 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskxPorqF/build/debug/bin/kudu as pid 30369
02:03:35.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.500792 30358 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-0/wal/instance
02:03:35.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.501263 30436 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
02:03:35.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.502223 30358 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskxPorqF/build/debug/bin/kudu
02:03:35.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskxPorqF/build/debug/bin/kudu
02:03:35.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-1/wal
02:03:35.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-1/data
02:03:35.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:03:35.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:03:35.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:03:35.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:03:35.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:03:35.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:03:35.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-1/logs
02:03:35.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-1/data/info.pb
02:03:35.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:03:35.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:03:35.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:03:35.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:03:35.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:03:35.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:03:35.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:03:35.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:03:35.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
02:03:35.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:03:35.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
02:03:35.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
02:03:35.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.165.189:39733
02:03:35.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.165.189
02:03:35.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:03:35.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.165.148:38003
02:03:35.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:03:35.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:03:35.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:03:35.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
02:03:35.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.29.165.190:39217,127.29.165.189:39733,127.29.165.188:38583
02:03:35.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
02:03:35.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:43275/jwks.json with env {}
02:03:35.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.506829 30436 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.165.190" port: 39217 } has no permanent_uuid. Determining permanent_uuid...
02:03:35.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.522096 30436 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.165.189" port: 39733 } has no permanent_uuid. Determining permanent_uuid...
02:03:35.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.522961 30386 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.165.189:39733: connect: Connection refused (error 111)
02:03:35.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.523834 30436 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.29.165.189:39733: Network error: Client connection negotiation failed: client connection to 127.29.165.189:39733: connect: Connection refused (error 111)
02:03:35.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.573108 30436 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.165.189" port: 39733 } attempt: 1
02:03:35.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.574986 30436 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.29.165.189:39733: Network error: Client connection negotiation failed: client connection to 127.29.165.189:39733: connect: Connection refused (error 111)
02:03:35.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.602191 30438 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:03:35.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.602365 30438 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:03:35.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.602401 30438 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:03:35.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.605652 30438 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
02:03:35.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.605702 30438 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:03:35.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.605720 30438 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
02:03:35.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.605737 30438 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
02:03:35.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.605753 30438 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
02:03:35.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.605767 30438 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:43275/jwks.json
02:03:35.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.609524 30438 master_runner.cc:387] Master server non-default flags:
02:03:35.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:03:35.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.165.148:38003
02:03:35.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:03:35.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:03:35.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-1/data
02:03:35.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-1/wal
02:03:35.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
02:03:35.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.29.165.190:39217,127.29.165.189:39733,127.29.165.188:38583
02:03:35.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:03:35.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:03:35.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
02:03:35.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.165.189:39733
02:03:35.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
02:03:35.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:03:35.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
02:03:35.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:43275/jwks.json
02:03:35.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:03:35.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:03:35.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-1/data/info.pb
02:03:35.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.165.189
02:03:35.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:03:35.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:03:35.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.30438
02:03:35.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:03:35.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:03:35.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:03:35.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:03:35.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-1/logs
02:03:35.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:03:35.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:03:35.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:03:35.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
02:03:35.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:03:35.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
02:03:35.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type DEBUG
02:03:35.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 01:43:15 UTC on bdcb31816ec0
02:03:35.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11948
02:03:35.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.610417 30438 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:03:35.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.611441 30438 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:03:35.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.616950 30449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:35.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.617159 30446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:35.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.617202 30447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:35.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.618333 30438 server_base.cc:1061] running on GCE node
02:03:35.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.618857 30438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:03:35.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.619859 30438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:03:35.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.621070 30438 hybrid_clock.cc:648] HybridClock initialized: now 1779069815621062 us; error 31 us; skew 500 ppm
02:03:35.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.622867 30438 webserver.cc:533] Webserver started at http://127.29.165.189:37455/ using document root <none> and password file <none>
02:03:35.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.623472 30438 fs_manager.cc:362] Metadata directory not provided
02:03:35.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.623548 30438 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:03:35.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.623764 30438 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
02:03:35.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.625370 30438 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-1/data/instance:
02:03:35.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "62a54aff7f4a46a9af3cc1f2d0e7d25b"
02:03:35.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:03:35 on dist-test-slave-838s"
02:03:35.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.625905 30438 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-1/wal/instance:
02:03:35.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "62a54aff7f4a46a9af3cc1f2d0e7d25b"
02:03:35.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:03:35 on dist-test-slave-838s"
02:03:35.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.629411 30438 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.006s	sys 0.000s
02:03:35.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.631672 30455 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:03:35.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.632823 30438 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
02:03:35.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.632931 30438 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-1/data,/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-1/wal
02:03:35.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "62a54aff7f4a46a9af3cc1f2d0e7d25b"
02:03:35.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:03:35 on dist-test-slave-838s"
02:03:35.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.633023 30438 fs_report.cc:389] FS layout report
02:03:35.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:03:35.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-1/wal
02:03:35.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-1/wal
02:03:35.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-1/data/data
02:03:35.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:03:35.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:03:35.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:03:35.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:03:35.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:03:35.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:03:35.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:03:35.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:03:35.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:03:35.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:03:35.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:03:35.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.643162 30436 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.165.189" port: 39733 } attempt: 2
02:03:35.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.644555 30436 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.29.165.189:39733: Network error: Client connection negotiation failed: client connection to 127.29.165.189:39733: connect: Connection refused (error 111)
02:03:35.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.652608 30438 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:03:35.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.653199 30438 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:03:35.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.653339 30438 kserver.cc:163] Server-wide thread pool size limit: 3276
02:03:35.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.661427 30438 rpc_server.cc:307] RPC server started. Bound to: 127.29.165.189:39733
02:03:35.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.661434 30507 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.165.189:39733 every 8 connection(s)
02:03:35.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.662370 30438 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-1/data/info.pb
02:03:35.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.665128 30508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
02:03:35.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.667852 30358 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskxPorqF/build/debug/bin/kudu as pid 30438
02:03:35.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.667963 30358 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-1/wal/instance
02:03:35.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.668638 30358 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskxPorqF/build/debug/bin/kudu
02:03:35.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskxPorqF/build/debug/bin/kudu
02:03:35.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-2/wal
02:03:35.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-2/data
02:03:35.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:03:35.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:03:35.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:03:35.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:03:35.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:03:35.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:03:35.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-2/logs
02:03:35.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-2/data/info.pb
02:03:35.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:03:35.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:03:35.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:03:35.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:03:35.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:03:35.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:03:35.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:03:35.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:03:35.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
02:03:35.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:03:35.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
02:03:35.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
02:03:35.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.165.188:38583
02:03:35.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.165.188
02:03:35.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:03:35.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.165.148:38003
02:03:35.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:03:35.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:03:35.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:03:35.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
02:03:35.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.29.165.190:39217,127.29.165.189:39733,127.29.165.188:38583
02:03:35.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
02:03:35.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:43275/jwks.json with env {}
02:03:35.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.668999 30508 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.165.190" port: 39217 } has no permanent_uuid. Determining permanent_uuid...
02:03:35.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.679078 30508 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.165.189" port: 39733 } has no permanent_uuid. Determining permanent_uuid...
02:03:35.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.684590 30508 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.165.188" port: 38583 } has no permanent_uuid. Determining permanent_uuid...
02:03:35.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.685247 30458 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.165.188:38583: connect: Connection refused (error 111)
02:03:35.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.685973 30508 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.29.165.188:38583: Network error: Client connection negotiation failed: client connection to 127.29.165.188:38583: connect: Connection refused (error 111)
02:03:35.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.735169 30508 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.165.188" port: 38583 } attempt: 1
02:03:35.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.735733 30436 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.165.189" port: 39733 } attempt: 3
02:03:35.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.737192 30508 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.29.165.188:38583: Network error: Client connection negotiation failed: client connection to 127.29.165.188:38583: connect: Connection refused (error 111)
02:03:35.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.740244 30436 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.165.188" port: 38583 } has no permanent_uuid. Determining permanent_uuid...
02:03:35.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.741530 30436 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.29.165.188:38583: Network error: Client connection negotiation failed: client connection to 127.29.165.188:38583: connect: Connection refused (error 111)
02:03:35.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.770922 30510 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:03:35.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.771121 30510 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:03:35.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.771163 30510 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:03:35.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.772701 30436 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.165.188" port: 38583 } attempt: 1
02:03:35.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.774216 30436 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.29.165.188:38583: Network error: Client connection negotiation failed: client connection to 127.29.165.188:38583: connect: Connection refused (error 111)
02:03:35.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.774670 30510 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
02:03:35.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.774760 30510 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:03:35.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.774791 30510 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
02:03:35.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.774809 30510 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
02:03:35.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.774825 30510 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
02:03:35.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.774840 30510 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:43275/jwks.json
02:03:35.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.778365 30510 master_runner.cc:387] Master server non-default flags:
02:03:35.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:03:35.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.165.148:38003
02:03:35.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:03:35.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:03:35.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-2/data
02:03:35.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-2/wal
02:03:35.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
02:03:35.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.29.165.190:39217,127.29.165.189:39733,127.29.165.188:38583
02:03:35.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:03:35.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:03:35.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
02:03:35.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.165.188:38583
02:03:35.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
02:03:35.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:03:35.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
02:03:35.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:43275/jwks.json
02:03:35.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:03:35.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:03:35.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-2/data/info.pb
02:03:35.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.165.188
02:03:35.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:03:35.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:03:35.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.30510
02:03:35.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:03:35.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:03:35.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:03:35.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:03:35.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-2/logs
02:03:35.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:03:35.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:03:35.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:03:35.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
02:03:35.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:03:35.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
02:03:35.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type DEBUG
02:03:35.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 01:43:15 UTC on bdcb31816ec0
02:03:35.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11948
02:03:35.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.779531 30510 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:03:35.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.780813 30510 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:03:35.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.786687 30521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:35.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.786780 30518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:35.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.786850 30519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:35.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.788395 30510 server_base.cc:1061] running on GCE node
02:03:35.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.788930 30510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:03:35.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.789949 30510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:03:35.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.791131 30510 hybrid_clock.cc:648] HybridClock initialized: now 1779069815791081 us; error 65 us; skew 500 ppm
02:03:35.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.792690 30510 webserver.cc:533] Webserver started at http://127.29.165.188:45031/ using document root <none> and password file <none>
02:03:35.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.793346 30510 fs_manager.cc:362] Metadata directory not provided
02:03:35.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.793424 30510 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:03:35.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.793627 30510 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
02:03:35.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.795272 30510 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-2/data/instance:
02:03:35.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "fda1f9879cb84f22910837d1419bcc97"
02:03:35.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:03:35 on dist-test-slave-838s"
02:03:35.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.795653 30510 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-2/wal/instance:
02:03:35.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "fda1f9879cb84f22910837d1419bcc97"
02:03:35.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:03:35 on dist-test-slave-838s"
02:03:35.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.798595 30510 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.003s	sys 0.000s
02:03:35.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.800596 30527 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:03:35.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.801481 30510 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
02:03:35.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.801599 30510 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-2/data,/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-2/wal
02:03:35.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "fda1f9879cb84f22910837d1419bcc97"
02:03:35.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:03:35 on dist-test-slave-838s"
02:03:35.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.801687 30510 fs_report.cc:389] FS layout report
02:03:35.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:03:35.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-2/wal
02:03:35.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-2/wal
02:03:35.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-2/data/data
02:03:35.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:03:35.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:03:35.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:03:35.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:03:35.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:03:35.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:03:35.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:03:35.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:03:35.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:03:35.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:03:35.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:03:35.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.805370 30508 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.165.188" port: 38583 } attempt: 2
02:03:35.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.806610 30508 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.29.165.188:38583: Network error: Client connection negotiation failed: client connection to 127.29.165.188:38583: connect: Connection refused (error 111)
02:03:35.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.824896 30510 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:03:35.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.825626 30510 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:03:35.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.825773 30510 kserver.cc:163] Server-wide thread pool size limit: 3276
02:03:35.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.832902 30579 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.165.188:38583 every 8 connection(s)
02:03:35.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.832912 30510 rpc_server.cc:307] RPC server started. Bound to: 127.29.165.188:38583
02:03:35.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.833887 30510 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-2/data/info.pb
02:03:35.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.835536 30358 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskxPorqF/build/debug/bin/kudu as pid 30510
02:03:35.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.835638 30358 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/master-2/wal/instance
02:03:35.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.836874 30580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
02:03:35.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.840924 30580 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.165.190" port: 39217 } has no permanent_uuid. Determining permanent_uuid...
02:03:35.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.848934 30580 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.165.189" port: 39733 } has no permanent_uuid. Determining permanent_uuid...
02:03:35.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.849515 30436 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.165.188" port: 38583 } attempt: 2
02:03:35.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.854452 30580 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.165.188" port: 38583 } has no permanent_uuid. Determining permanent_uuid...
02:03:35.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.857944 30436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 324afe9044bf40299572cad6ecfc4d5d: Bootstrap starting.
02:03:35.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.859748 30580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fda1f9879cb84f22910837d1419bcc97: Bootstrap starting.
02:03:35.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.860365 30436 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 324afe9044bf40299572cad6ecfc4d5d: Neither blocks nor log segments found. Creating new log.
02:03:35.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.861156 30436 log.cc:826] T 00000000000000000000000000000000 P 324afe9044bf40299572cad6ecfc4d5d: Log is configured to *not* fsync() on all Append() calls
02:03:35.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.861526 30580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fda1f9879cb84f22910837d1419bcc97: Neither blocks nor log segments found. Creating new log.
02:03:35.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.862022 30580 log.cc:826] T 00000000000000000000000000000000 P fda1f9879cb84f22910837d1419bcc97: Log is configured to *not* fsync() on all Append() calls
02:03:35.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.862686 30436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 324afe9044bf40299572cad6ecfc4d5d: No bootstrap required, opened a new log
02:03:35.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.864164 30580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fda1f9879cb84f22910837d1419bcc97: No bootstrap required, opened a new log
02:03:35.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.865728 30436 raft_consensus.cc:359] T 00000000000000000000000000000000 P 324afe9044bf40299572cad6ecfc4d5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "324afe9044bf40299572cad6ecfc4d5d" member_type: VOTER last_known_addr { host: "127.29.165.190" port: 39217 } } peers { permanent_uuid: "62a54aff7f4a46a9af3cc1f2d0e7d25b" member_type: VOTER last_known_addr { host: "127.29.165.189" port: 39733 } } peers { permanent_uuid: "fda1f9879cb84f22910837d1419bcc97" member_type: VOTER last_known_addr { host: "127.29.165.188" port: 38583 } }
02:03:35.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.865864 30436 raft_consensus.cc:385] T 00000000000000000000000000000000 P 324afe9044bf40299572cad6ecfc4d5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
02:03:35.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.865901 30436 raft_consensus.cc:740] T 00000000000000000000000000000000 P 324afe9044bf40299572cad6ecfc4d5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 324afe9044bf40299572cad6ecfc4d5d, State: Initialized, Role: FOLLOWER
02:03:35.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.866246 30436 consensus_queue.cc:260] T 00000000000000000000000000000000 P 324afe9044bf40299572cad6ecfc4d5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "324afe9044bf40299572cad6ecfc4d5d" member_type: VOTER last_known_addr { host: "127.29.165.190" port: 39217 } } peers { permanent_uuid: "62a54aff7f4a46a9af3cc1f2d0e7d25b" member_type: VOTER last_known_addr { host: "127.29.165.189" port: 39733 } } peers { permanent_uuid: "fda1f9879cb84f22910837d1419bcc97" member_type: VOTER last_known_addr { host: "127.29.165.188" port: 38583 } }
02:03:35.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.866616 30589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324afe9044bf40299572cad6ecfc4d5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "324afe9044bf40299572cad6ecfc4d5d" member_type: VOTER last_known_addr { host: "127.29.165.190" port: 39217 } } peers { permanent_uuid: "62a54aff7f4a46a9af3cc1f2d0e7d25b" member_type: VOTER last_known_addr { host: "127.29.165.189" port: 39733 } } peers { permanent_uuid: "fda1f9879cb84f22910837d1419bcc97" member_type: VOTER last_known_addr { host: "127.29.165.188" port: 38583 } } }
02:03:35.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.866747 30589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324afe9044bf40299572cad6ecfc4d5d [sys.catalog]: This master's current role is: FOLLOWER
02:03:35.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.867136 30580 raft_consensus.cc:359] T 00000000000000000000000000000000 P fda1f9879cb84f22910837d1419bcc97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "324afe9044bf40299572cad6ecfc4d5d" member_type: VOTER last_known_addr { host: "127.29.165.190" port: 39217 } } peers { permanent_uuid: "62a54aff7f4a46a9af3cc1f2d0e7d25b" member_type: VOTER last_known_addr { host: "127.29.165.189" port: 39733 } } peers { permanent_uuid: "fda1f9879cb84f22910837d1419bcc97" member_type: VOTER last_known_addr { host: "127.29.165.188" port: 38583 } }
02:03:35.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.867274 30436 sys_catalog.cc:565] T 00000000000000000000000000000000 P 324afe9044bf40299572cad6ecfc4d5d [sys.catalog]: configured and running, proceeding with master startup.
02:03:35.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.867303 30580 raft_consensus.cc:385] T 00000000000000000000000000000000 P fda1f9879cb84f22910837d1419bcc97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
02:03:35.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.867360 30580 raft_consensus.cc:740] T 00000000000000000000000000000000 P fda1f9879cb84f22910837d1419bcc97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fda1f9879cb84f22910837d1419bcc97, State: Initialized, Role: FOLLOWER
02:03:35.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.868783 30580 consensus_queue.cc:260] T 00000000000000000000000000000000 P fda1f9879cb84f22910837d1419bcc97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "324afe9044bf40299572cad6ecfc4d5d" member_type: VOTER last_known_addr { host: "127.29.165.190" port: 39217 } } peers { permanent_uuid: "62a54aff7f4a46a9af3cc1f2d0e7d25b" member_type: VOTER last_known_addr { host: "127.29.165.189" port: 39733 } } peers { permanent_uuid: "fda1f9879cb84f22910837d1419bcc97" member_type: VOTER last_known_addr { host: "127.29.165.188" port: 38583 } }
02:03:35.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.870118 30580 sys_catalog.cc:565] T 00000000000000000000000000000000 P fda1f9879cb84f22910837d1419bcc97 [sys.catalog]: configured and running, proceeding with master startup.
02:03:35.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.872157 30601 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 324afe9044bf40299572cad6ecfc4d5d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
02:03:35.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.872231 30601 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
02:03:35.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.873589 30369 master_runner.cc:187] Error getting master registration for 127.29.165.189:39733: OK, instance_id { permanent_uuid: "62a54aff7f4a46a9af3cc1f2d0e7d25b" instance_seqno: 1779069815659141 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
02:03:35.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.875715 30369 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
02:03:35.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.876317 30510 master_runner.cc:187] Error getting master registration for 127.29.165.189:39733: OK, instance_id { permanent_uuid: "62a54aff7f4a46a9af3cc1f2d0e7d25b" instance_seqno: 1779069815659141 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
02:03:35.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.878458 30510 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
02:03:35.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.880580 30613 catalog_manager.cc:1593] T 00000000000000000000000000000000 P fda1f9879cb84f22910837d1419bcc97: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
02:03:35.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.880658 30613 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
02:03:35.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.880887 30600 sys_catalog.cc:455] T 00000000000000000000000000000000 P fda1f9879cb84f22910837d1419bcc97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "324afe9044bf40299572cad6ecfc4d5d" member_type: VOTER last_known_addr { host: "127.29.165.190" port: 39217 } } peers { permanent_uuid: "62a54aff7f4a46a9af3cc1f2d0e7d25b" member_type: VOTER last_known_addr { host: "127.29.165.189" port: 39733 } } peers { permanent_uuid: "fda1f9879cb84f22910837d1419bcc97" member_type: VOTER last_known_addr { host: "127.29.165.188" port: 38583 } } }
02:03:35.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.880986 30600 sys_catalog.cc:458] T 00000000000000000000000000000000 P fda1f9879cb84f22910837d1419bcc97 [sys.catalog]: This master's current role is: FOLLOWER
02:03:35.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.892977 30600 raft_consensus.cc:493] T 00000000000000000000000000000000 P fda1f9879cb84f22910837d1419bcc97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
02:03:35.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.893136 30600 raft_consensus.cc:515] T 00000000000000000000000000000000 P fda1f9879cb84f22910837d1419bcc97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "324afe9044bf40299572cad6ecfc4d5d" member_type: VOTER last_known_addr { host: "127.29.165.190" port: 39217 } } peers { permanent_uuid: "62a54aff7f4a46a9af3cc1f2d0e7d25b" member_type: VOTER last_known_addr { host: "127.29.165.189" port: 39733 } } peers { permanent_uuid: "fda1f9879cb84f22910837d1419bcc97" member_type: VOTER last_known_addr { host: "127.29.165.188" port: 38583 } }
02:03:35.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.894308 30410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fda1f9879cb84f22910837d1419bcc97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "324afe9044bf40299572cad6ecfc4d5d" is_pre_election: true
02:03:35.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.894352 30600 leader_election.cc:290] T 00000000000000000000000000000000 P fda1f9879cb84f22910837d1419bcc97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 324afe9044bf40299572cad6ecfc4d5d (127.29.165.190:39217), 62a54aff7f4a46a9af3cc1f2d0e7d25b (127.29.165.189:39733)
02:03:35.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.894519 30410 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 324afe9044bf40299572cad6ecfc4d5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fda1f9879cb84f22910837d1419bcc97 in term 0.
02:03:35.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.894912 30531 leader_election.cc:304] T 00000000000000000000000000000000 P fda1f9879cb84f22910837d1419bcc97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 324afe9044bf40299572cad6ecfc4d5d, fda1f9879cb84f22910837d1419bcc97; no voters: 
02:03:35.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.895293 30600 raft_consensus.cc:2804] T 00000000000000000000000000000000 P fda1f9879cb84f22910837d1419bcc97 [term 0 FOLLOWER]: Leader pre-election won for term 1
02:03:35.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.895371 30600 raft_consensus.cc:493] T 00000000000000000000000000000000 P fda1f9879cb84f22910837d1419bcc97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
02:03:35.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.895416 30600 raft_consensus.cc:3060] T 00000000000000000000000000000000 P fda1f9879cb84f22910837d1419bcc97 [term 0 FOLLOWER]: Advancing to term 1
02:03:35.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.896526 30600 raft_consensus.cc:515] T 00000000000000000000000000000000 P fda1f9879cb84f22910837d1419bcc97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "324afe9044bf40299572cad6ecfc4d5d" member_type: VOTER last_known_addr { host: "127.29.165.190" port: 39217 } } peers { permanent_uuid: "62a54aff7f4a46a9af3cc1f2d0e7d25b" member_type: VOTER last_known_addr { host: "127.29.165.189" port: 39733 } } peers { permanent_uuid: "fda1f9879cb84f22910837d1419bcc97" member_type: VOTER last_known_addr { host: "127.29.165.188" port: 38583 } }
02:03:35.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.897109 30410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fda1f9879cb84f22910837d1419bcc97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "324afe9044bf40299572cad6ecfc4d5d"
02:03:35.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.897210 30410 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 324afe9044bf40299572cad6ecfc4d5d [term 0 FOLLOWER]: Advancing to term 1
02:03:35.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.897337 30600 leader_election.cc:290] T 00000000000000000000000000000000 P fda1f9879cb84f22910837d1419bcc97 [CANDIDATE]: Term 1 election: Requested vote from peers 324afe9044bf40299572cad6ecfc4d5d (127.29.165.190:39217), 62a54aff7f4a46a9af3cc1f2d0e7d25b (127.29.165.189:39733)
02:03:35.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.897815 30508 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.165.188" port: 38583 } attempt: 3
02:03:35.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.898322 30410 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 324afe9044bf40299572cad6ecfc4d5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fda1f9879cb84f22910837d1419bcc97 in term 1.
02:03:35.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.898842 30531 leader_election.cc:304] T 00000000000000000000000000000000 P fda1f9879cb84f22910837d1419bcc97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 324afe9044bf40299572cad6ecfc4d5d, fda1f9879cb84f22910837d1419bcc97; no voters: 
02:03:35.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.898984 30600 raft_consensus.cc:2804] T 00000000000000000000000000000000 P fda1f9879cb84f22910837d1419bcc97 [term 1 FOLLOWER]: Leader election won for term 1
02:03:35.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.899156 30600 raft_consensus.cc:697] T 00000000000000000000000000000000 P fda1f9879cb84f22910837d1419bcc97 [term 1 LEADER]: Becoming Leader. State: Replica: fda1f9879cb84f22910837d1419bcc97, State: Running, Role: LEADER
02:03:35.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.899425 30600 consensus_queue.cc:237] T 00000000000000000000000000000000 P fda1f9879cb84f22910837d1419bcc97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "324afe9044bf40299572cad6ecfc4d5d" member_type: VOTER last_known_addr { host: "127.29.165.190" port: 39217 } } peers { permanent_uuid: "62a54aff7f4a46a9af3cc1f2d0e7d25b" member_type: VOTER last_known_addr { host: "127.29.165.189" port: 39733 } } peers { permanent_uuid: "fda1f9879cb84f22910837d1419bcc97" member_type: VOTER last_known_addr { host: "127.29.165.188" port: 38583 } }
02:03:35.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.901571 30615 sys_catalog.cc:455] T 00000000000000000000000000000000 P fda1f9879cb84f22910837d1419bcc97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fda1f9879cb84f22910837d1419bcc97. Latest consensus state: current_term: 1 leader_uuid: "fda1f9879cb84f22910837d1419bcc97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "324afe9044bf40299572cad6ecfc4d5d" member_type: VOTER last_known_addr { host: "127.29.165.190" port: 39217 } } peers { permanent_uuid: "62a54aff7f4a46a9af3cc1f2d0e7d25b" member_type: VOTER last_known_addr { host: "127.29.165.189" port: 39733 } } peers { permanent_uuid: "fda1f9879cb84f22910837d1419bcc97" member_type: VOTER last_known_addr { host: "127.29.165.188" port: 38583 } } }
02:03:35.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.901746 30615 sys_catalog.cc:458] T 00000000000000000000000000000000 P fda1f9879cb84f22910837d1419bcc97 [sys.catalog]: This master's current role is: LEADER
02:03:35.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.903947 30617 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
02:03:35.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.905151 30508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62a54aff7f4a46a9af3cc1f2d0e7d25b: Bootstrap starting.
02:03:35.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.905642 30617 catalog_manager.cc:1519] Initializing Kudu cluster ID...
02:03:35.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.910009 30508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62a54aff7f4a46a9af3cc1f2d0e7d25b: Neither blocks nor log segments found. Creating new log.
02:03:35.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.910821 30508 log.cc:826] T 00000000000000000000000000000000 P 62a54aff7f4a46a9af3cc1f2d0e7d25b: Log is configured to *not* fsync() on all Append() calls
02:03:35.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.916844 30508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62a54aff7f4a46a9af3cc1f2d0e7d25b: No bootstrap required, opened a new log
02:03:35.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.925194 30508 raft_consensus.cc:359] T 00000000000000000000000000000000 P 62a54aff7f4a46a9af3cc1f2d0e7d25b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "324afe9044bf40299572cad6ecfc4d5d" member_type: VOTER last_known_addr { host: "127.29.165.190" port: 39217 } } peers { permanent_uuid: "62a54aff7f4a46a9af3cc1f2d0e7d25b" member_type: VOTER last_known_addr { host: "127.29.165.189" port: 39733 } } peers { permanent_uuid: "fda1f9879cb84f22910837d1419bcc97" member_type: VOTER last_known_addr { host: "127.29.165.188" port: 38583 } }
02:03:35.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.925803 30508 raft_consensus.cc:385] T 00000000000000000000000000000000 P 62a54aff7f4a46a9af3cc1f2d0e7d25b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
02:03:35.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.926220 30508 raft_consensus.cc:740] T 00000000000000000000000000000000 P 62a54aff7f4a46a9af3cc1f2d0e7d25b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62a54aff7f4a46a9af3cc1f2d0e7d25b, State: Initialized, Role: FOLLOWER
02:03:35.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.927278 30508 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62a54aff7f4a46a9af3cc1f2d0e7d25b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "324afe9044bf40299572cad6ecfc4d5d" member_type: VOTER last_known_addr { host: "127.29.165.190" port: 39217 } } peers { permanent_uuid: "62a54aff7f4a46a9af3cc1f2d0e7d25b" member_type: VOTER last_known_addr { host: "127.29.165.189" port: 39733 } } peers { permanent_uuid: "fda1f9879cb84f22910837d1419bcc97" member_type: VOTER last_known_addr { host: "127.29.165.188" port: 38583 } }
02:03:35.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.929445 30620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62a54aff7f4a46a9af3cc1f2d0e7d25b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "324afe9044bf40299572cad6ecfc4d5d" member_type: VOTER last_known_addr { host: "127.29.165.190" port: 39217 } } peers { permanent_uuid: "62a54aff7f4a46a9af3cc1f2d0e7d25b" member_type: VOTER last_known_addr { host: "127.29.165.189" port: 39733 } } peers { permanent_uuid: "fda1f9879cb84f22910837d1419bcc97" member_type: VOTER last_known_addr { host: "127.29.165.188" port: 38583 } } }
02:03:35.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.929695 30410 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 324afe9044bf40299572cad6ecfc4d5d [term 1 FOLLOWER]: Refusing update from remote peer fda1f9879cb84f22910837d1419bcc97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
02:03:35.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.930047 30620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62a54aff7f4a46a9af3cc1f2d0e7d25b [sys.catalog]: This master's current role is: FOLLOWER
02:03:35.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.930352 30615 consensus_queue.cc:1048] T 00000000000000000000000000000000 P fda1f9879cb84f22910837d1419bcc97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "324afe9044bf40299572cad6ecfc4d5d" member_type: VOTER last_known_addr { host: "127.29.165.190" port: 39217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
02:03:35.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.931283 30480 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 62a54aff7f4a46a9af3cc1f2d0e7d25b [term 0 FOLLOWER]: Advancing to term 1
02:03:35.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.931811 30481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fda1f9879cb84f22910837d1419bcc97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62a54aff7f4a46a9af3cc1f2d0e7d25b"
02:03:35.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.932459 30480 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 62a54aff7f4a46a9af3cc1f2d0e7d25b [term 1 FOLLOWER]: Refusing update from remote peer fda1f9879cb84f22910837d1419bcc97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
02:03:35.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.932511 30482 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fda1f9879cb84f22910837d1419bcc97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62a54aff7f4a46a9af3cc1f2d0e7d25b" is_pre_election: true
02:03:35.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.932962 30508 sys_catalog.cc:565] T 00000000000000000000000000000000 P 62a54aff7f4a46a9af3cc1f2d0e7d25b [sys.catalog]: configured and running, proceeding with master startup.
02:03:35.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.933024 30615 consensus_queue.cc:1048] T 00000000000000000000000000000000 P fda1f9879cb84f22910837d1419bcc97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62a54aff7f4a46a9af3cc1f2d0e7d25b" member_type: VOTER last_known_addr { host: "127.29.165.189" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
02:03:35.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.938485 30589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324afe9044bf40299572cad6ecfc4d5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader fda1f9879cb84f22910837d1419bcc97. Latest consensus state: current_term: 1 leader_uuid: "fda1f9879cb84f22910837d1419bcc97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "324afe9044bf40299572cad6ecfc4d5d" member_type: VOTER last_known_addr { host: "127.29.165.190" port: 39217 } } peers { permanent_uuid: "62a54aff7f4a46a9af3cc1f2d0e7d25b" member_type: VOTER last_known_addr { host: "127.29.165.189" port: 39733 } } peers { permanent_uuid: "fda1f9879cb84f22910837d1419bcc97" member_type: VOTER last_known_addr { host: "127.29.165.188" port: 38583 } } }
02:03:35.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.938606 30589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324afe9044bf40299572cad6ecfc4d5d [sys.catalog]: This master's current role is: FOLLOWER
02:03:35.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.946785 30620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62a54aff7f4a46a9af3cc1f2d0e7d25b [sys.catalog]: SysCatalogTable state changed. Reason: New leader fda1f9879cb84f22910837d1419bcc97. Latest consensus state: current_term: 1 leader_uuid: "fda1f9879cb84f22910837d1419bcc97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "324afe9044bf40299572cad6ecfc4d5d" member_type: VOTER last_known_addr { host: "127.29.165.190" port: 39217 } } peers { permanent_uuid: "62a54aff7f4a46a9af3cc1f2d0e7d25b" member_type: VOTER last_known_addr { host: "127.29.165.189" port: 39733 } } peers { permanent_uuid: "fda1f9879cb84f22910837d1419bcc97" member_type: VOTER last_known_addr { host: "127.29.165.188" port: 38583 } } }
02:03:35.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.947214 30620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62a54aff7f4a46a9af3cc1f2d0e7d25b [sys.catalog]: This master's current role is: FOLLOWER
02:03:35.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.949694 30615 sys_catalog.cc:455] T 00000000000000000000000000000000 P fda1f9879cb84f22910837d1419bcc97 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fda1f9879cb84f22910837d1419bcc97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "324afe9044bf40299572cad6ecfc4d5d" member_type: VOTER last_known_addr { host: "127.29.165.190" port: 39217 } } peers { permanent_uuid: "62a54aff7f4a46a9af3cc1f2d0e7d25b" member_type: VOTER last_known_addr { host: "127.29.165.189" port: 39733 } } peers { permanent_uuid: "fda1f9879cb84f22910837d1419bcc97" member_type: VOTER last_known_addr { host: "127.29.165.188" port: 38583 } } }
02:03:35.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.949802 30615 sys_catalog.cc:458] T 00000000000000000000000000000000 P fda1f9879cb84f22910837d1419bcc97 [sys.catalog]: This master's current role is: LEADER
02:03:35.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.949905 30618 mvcc.cc:204] Tried to move back new op lower bound from 7287069966030233600 to 7287069965926641664. Current Snapshot: MvccSnapshot[applied={T|T < 7287069966030233600}]
02:03:35.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.952610 30620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62a54aff7f4a46a9af3cc1f2d0e7d25b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fda1f9879cb84f22910837d1419bcc97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "324afe9044bf40299572cad6ecfc4d5d" member_type: VOTER last_known_addr { host: "127.29.165.190" port: 39217 } } peers { permanent_uuid: "62a54aff7f4a46a9af3cc1f2d0e7d25b" member_type: VOTER last_known_addr { host: "127.29.165.189" port: 39733 } } peers { permanent_uuid: "fda1f9879cb84f22910837d1419bcc97" member_type: VOTER last_known_addr { host: "127.29.165.188" port: 38583 } } }
02:03:35.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.952719 30620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62a54aff7f4a46a9af3cc1f2d0e7d25b [sys.catalog]: This master's current role is: FOLLOWER
02:03:35.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.955026 30600 sys_catalog.cc:455] T 00000000000000000000000000000000 P fda1f9879cb84f22910837d1419bcc97 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fda1f9879cb84f22910837d1419bcc97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "324afe9044bf40299572cad6ecfc4d5d" member_type: VOTER last_known_addr { host: "127.29.165.190" port: 39217 } } peers { permanent_uuid: "62a54aff7f4a46a9af3cc1f2d0e7d25b" member_type: VOTER last_known_addr { host: "127.29.165.189" port: 39733 } } peers { permanent_uuid: "fda1f9879cb84f22910837d1419bcc97" member_type: VOTER last_known_addr { host: "127.29.165.188" port: 38583 } } }
02:03:35.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.955125 30600 sys_catalog.cc:458] T 00000000000000000000000000000000 P fda1f9879cb84f22910837d1419bcc97 [sys.catalog]: This master's current role is: LEADER
02:03:35.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.955984 30589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324afe9044bf40299572cad6ecfc4d5d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fda1f9879cb84f22910837d1419bcc97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "324afe9044bf40299572cad6ecfc4d5d" member_type: VOTER last_known_addr { host: "127.29.165.190" port: 39217 } } peers { permanent_uuid: "62a54aff7f4a46a9af3cc1f2d0e7d25b" member_type: VOTER last_known_addr { host: "127.29.165.189" port: 39733 } } peers { permanent_uuid: "fda1f9879cb84f22910837d1419bcc97" member_type: VOTER last_known_addr { host: "127.29.165.188" port: 38583 } } }
02:03:35.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.956089 30589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324afe9044bf40299572cad6ecfc4d5d [sys.catalog]: This master's current role is: FOLLOWER
02:03:35.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.957656 30634 mvcc.cc:204] Tried to move back new op lower bound from 7287069966030233600 to 7287069965926641664. Current Snapshot: MvccSnapshot[applied={T|T < 7287069966030233600}]
02:03:35.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.958230 30632 mvcc.cc:204] Tried to move back new op lower bound from 7287069966030233600 to 7287069965926641664. Current Snapshot: MvccSnapshot[applied={T|T < 7287069966030233600}]
02:03:35.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.959985 30617 catalog_manager.cc:1382] Generated new cluster ID: f4a37041a44a4514a7b70d9da292b44a
02:03:35.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.960047 30617 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
02:03:35.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.963132 30631 catalog_manager.cc:1294] Loaded cluster ID: f4a37041a44a4514a7b70d9da292b44a
02:03:35.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.963174 30631 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 62a54aff7f4a46a9af3cc1f2d0e7d25b: loading cluster ID for follower catalog manager: success
02:03:35.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.963677 30631 catalog_manager.cc:1611] T 00000000000000000000000000000000 P 62a54aff7f4a46a9af3cc1f2d0e7d25b: acquiring CA information for follower catalog manager: Not found: root CA entry not found
02:03:35.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:35.963745 30631 catalog_manager.cc:883] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
02:03:35.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.976054 30617 catalog_manager.cc:1405] Generated new certificate authority record
02:03:35.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.977326 30617 catalog_manager.cc:1539] Loading token signing keys...
02:03:35.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.991261 30617 catalog_manager.cc:6069] T 00000000000000000000000000000000 P fda1f9879cb84f22910837d1419bcc97: Generated new TSK 0
02:03:35.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:35.992075 30617 catalog_manager.cc:1549] Initializing in-progress tserver states...
02:03:36.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.005445 30358 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskxPorqF/build/debug/bin/kudu
02:03:36.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskxPorqF/build/debug/bin/kudu
02:03:36.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/wal
02:03:36.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/data
02:03:36.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:03:36.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:03:36.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:03:36.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:03:36.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:03:36.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:03:36.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/logs
02:03:36.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/data/info.pb
02:03:36.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:03:36.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:03:36.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:03:36.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:03:36.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:03:36.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:03:36.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:03:36.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:03:36.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:03:36.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:03:36.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.165.129:0
02:03:36.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.165.129
02:03:36.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.165.129
02:03:36.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:03:36.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.165.190:39217,127.29.165.189:39733,127.29.165.188:38583
02:03:36.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.165.148:38003
02:03:36.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:03:36.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:03:36.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:03:36.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:36.156489 30640 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:03:36.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:36.156714 30640 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:03:36.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:36.156769 30640 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:03:36.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:36.160257 30640 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:03:36.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:36.160394 30640 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.165.129
02:03:36.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.164033 30640 tablet_server_runner.cc:78] Tablet server non-default flags:
02:03:36.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:03:36.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.165.148:38003
02:03:36.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:03:36.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:03:36.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/data
02:03:36.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/wal
02:03:36.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:03:36.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:03:36.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.165.129:0
02:03:36.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:03:36.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:03:36.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:03:36.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/data/info.pb
02:03:36.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.165.129
02:03:36.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:03:36.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.165.190:39217,127.29.165.189:39733,127.29.165.188:38583
02:03:36.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:03:36.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.30640
02:03:36.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:03:36.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:03:36.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:03:36.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:03:36.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.165.129
02:03:36.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/logs
02:03:36.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:03:36.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:03:36.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:03:36.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:03:36.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:03:36.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
02:03:36.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type DEBUG
02:03:36.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 01:43:15 UTC on bdcb31816ec0
02:03:36.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11948
02:03:36.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.165323 30640 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:03:36.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.166826 30640 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:03:36.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:36.173162 30646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:36.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.173305 30640 server_base.cc:1061] running on GCE node
02:03:36.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:36.173065 30645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:36.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:36.173049 30648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:36.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.174259 30640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:03:36.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.174849 30640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:03:36.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.176005 30640 hybrid_clock.cc:648] HybridClock initialized: now 1779069816175953 us; error 67 us; skew 500 ppm
02:03:36.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.177712 30640 webserver.cc:533] Webserver started at http://127.29.165.129:46663/ using document root <none> and password file <none>
02:03:36.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.178260 30640 fs_manager.cc:362] Metadata directory not provided
02:03:36.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.178335 30640 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:03:36.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.178550 30640 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
02:03:36.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.180028 30640 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/data/instance:
02:03:36.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "06f0647651a749f8a533afd451fe5fcb"
02:03:36.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:03:36 on dist-test-slave-838s"
02:03:36.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.180521 30640 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/wal/instance:
02:03:36.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "06f0647651a749f8a533afd451fe5fcb"
02:03:36.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:03:36 on dist-test-slave-838s"
02:03:36.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.183532 30640 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.004s	sys 0.000s
02:03:36.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.185472 30654 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:03:36.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.186435 30640 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
02:03:36.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.186688 30640 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/data,/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/wal
02:03:36.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "06f0647651a749f8a533afd451fe5fcb"
02:03:36.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:03:36 on dist-test-slave-838s"
02:03:36.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.186821 30640 fs_report.cc:389] FS layout report
02:03:36.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:03:36.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/wal
02:03:36.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/wal
02:03:36.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/data/data
02:03:36.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:03:36.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:03:36.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:03:36.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:03:36.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:03:36.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:03:36.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:03:36.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:03:36.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:03:36.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:03:36.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:03:36.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.209100 30640 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:03:36.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.209842 30640 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:03:36.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.210036 30640 kserver.cc:163] Server-wide thread pool size limit: 3276
02:03:36.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.210587 30640 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:03:36.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.211566 30640 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
02:03:36.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.211627 30640 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
02:03:36.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.211678 30640 ts_tablet_manager.cc:616] Registered 0 tablets
02:03:36.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.211697 30640 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
02:03:36.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.220309 30640 rpc_server.cc:307] RPC server started. Bound to: 127.29.165.129:45737
02:03:36.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.220366 30767 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.165.129:45737 every 8 connection(s)
02:03:36.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.221261 30640 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/data/info.pb
02:03:36.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.222517 30358 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskxPorqF/build/debug/bin/kudu as pid 30640
02:03:36.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.222641 30358 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/wal/instance
02:03:36.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.225075 30358 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskxPorqF/build/debug/bin/kudu
02:03:36.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskxPorqF/build/debug/bin/kudu
02:03:36.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/wal
02:03:36.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/data
02:03:36.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:03:36.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:03:36.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:03:36.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:03:36.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:03:36.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:03:36.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/logs
02:03:36.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/data/info.pb
02:03:36.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:03:36.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:03:36.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:03:36.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:03:36.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:03:36.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:03:36.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:03:36.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:03:36.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:03:36.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:03:36.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.165.130:0
02:03:36.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.165.130
02:03:36.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.165.130
02:03:36.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:03:36.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.165.190:39217,127.29.165.189:39733,127.29.165.188:38583
02:03:36.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.165.148:38003
02:03:36.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:03:36.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:03:36.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:03:36.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.231208 30768 heartbeater.cc:344] Connected to a master server at 127.29.165.190:39217
02:03:36.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.231240 30770 heartbeater.cc:344] Connected to a master server at 127.29.165.189:39733
02:03:36.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.231437 30770 heartbeater.cc:461] Registering TS with master...
02:03:36.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.231437 30768 heartbeater.cc:461] Registering TS with master...
02:03:36.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.231986 30770 heartbeater.cc:507] Master 127.29.165.189:39733 requested a full tablet report, sending...
02:03:36.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.231986 30768 heartbeater.cc:507] Master 127.29.165.190:39217 requested a full tablet report, sending...
02:03:36.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.233430 30472 ts_manager.cc:194] Registered new tserver with Master: 06f0647651a749f8a533afd451fe5fcb (127.29.165.129:45737)
02:03:36.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.233568 30400 ts_manager.cc:194] Registered new tserver with Master: 06f0647651a749f8a533afd451fe5fcb (127.29.165.129:45737)
02:03:36.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.240121 30769 heartbeater.cc:344] Connected to a master server at 127.29.165.188:38583
02:03:36.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.240233 30769 heartbeater.cc:461] Registering TS with master...
02:03:36.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.240424 30769 heartbeater.cc:507] Master 127.29.165.188:38583 requested a full tablet report, sending...
02:03:36.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.241896 30544 ts_manager.cc:194] Registered new tserver with Master: 06f0647651a749f8a533afd451fe5fcb (127.29.165.129:45737)
02:03:36.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.243356 30544 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.165.129:36445
02:03:36.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:36.329846 30775 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:03:36.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:36.330081 30775 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:03:36.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:36.330135 30775 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:03:36.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:36.333410 30775 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:03:36.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:36.333534 30775 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.165.130
02:03:36.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.337273 30775 tablet_server_runner.cc:78] Tablet server non-default flags:
02:03:36.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:03:36.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.165.148:38003
02:03:36.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:03:36.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:03:36.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/data
02:03:36.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/wal
02:03:36.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:03:36.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:03:36.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.165.130:0
02:03:36.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:03:36.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:03:36.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:03:36.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/data/info.pb
02:03:36.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.165.130
02:03:36.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:03:36.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.165.190:39217,127.29.165.189:39733,127.29.165.188:38583
02:03:36.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:03:36.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.30775
02:03:36.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:03:36.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:03:36.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:03:36.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:03:36.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.165.130
02:03:36.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/logs
02:03:36.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:03:36.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:03:36.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:03:36.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:03:36.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:03:36.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
02:03:36.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type DEBUG
02:03:36.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 01:43:15 UTC on bdcb31816ec0
02:03:36.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11948
02:03:36.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.338563 30775 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:03:36.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.339897 30775 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:03:36.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:36.346632 30781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:36.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:36.346625 30780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:36.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:36.346626 30783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:36.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.346824 30775 server_base.cc:1061] running on GCE node
02:03:36.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.347652 30775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:03:36.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.348088 30775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:03:36.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.349231 30775 hybrid_clock.cc:648] HybridClock initialized: now 1779069816349215 us; error 34 us; skew 500 ppm
02:03:36.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.350857 30775 webserver.cc:533] Webserver started at http://127.29.165.130:33955/ using document root <none> and password file <none>
02:03:36.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.351357 30775 fs_manager.cc:362] Metadata directory not provided
02:03:36.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.351405 30775 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:03:36.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.351573 30775 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
02:03:36.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.352924 30775 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/data/instance:
02:03:36.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "4e39ec54c2d642feafa81dc5d4f066c2"
02:03:36.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:03:36 on dist-test-slave-838s"
02:03:36.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.353319 30775 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/wal/instance:
02:03:36.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "4e39ec54c2d642feafa81dc5d4f066c2"
02:03:36.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:03:36 on dist-test-slave-838s"
02:03:36.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.356427 30775 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.002s	sys 0.003s
02:03:36.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.358568 30789 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:03:36.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.359498 30775 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
02:03:36.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.359588 30775 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/data,/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/wal
02:03:36.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "4e39ec54c2d642feafa81dc5d4f066c2"
02:03:36.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:03:36 on dist-test-slave-838s"
02:03:36.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.359702 30775 fs_report.cc:389] FS layout report
02:03:36.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:03:36.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/wal
02:03:36.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/wal
02:03:36.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/data/data
02:03:36.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:03:36.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:03:36.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:03:36.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:03:36.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:03:36.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:03:36.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:03:36.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:03:36.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:03:36.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:03:36.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:03:36.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.369607 30775 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:03:36.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.370102 30775 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:03:36.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.370272 30775 kserver.cc:163] Server-wide thread pool size limit: 3276
02:03:36.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.370746 30775 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:03:36.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.371639 30775 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
02:03:36.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.371712 30775 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
02:03:36.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.371755 30775 ts_tablet_manager.cc:616] Registered 0 tablets
02:03:36.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.371769 30775 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
02:03:36.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.380383 30775 rpc_server.cc:307] RPC server started. Bound to: 127.29.165.130:35181
02:03:36.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.380470 30902 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.165.130:35181 every 8 connection(s)
02:03:36.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.381311 30775 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/data/info.pb
02:03:36.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.390082 30904 heartbeater.cc:344] Connected to a master server at 127.29.165.188:38583
02:03:36.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.390244 30903 heartbeater.cc:344] Connected to a master server at 127.29.165.190:39217
02:03:36.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.390308 30903 heartbeater.cc:461] Registering TS with master...
02:03:36.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.390620 30905 heartbeater.cc:344] Connected to a master server at 127.29.165.189:39733
02:03:36.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.390308 30904 heartbeater.cc:461] Registering TS with master...
02:03:36.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.390731 30905 heartbeater.cc:461] Registering TS with master...
02:03:36.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.391001 30905 heartbeater.cc:507] Master 127.29.165.189:39733 requested a full tablet report, sending...
02:03:36.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.391007 30904 heartbeater.cc:507] Master 127.29.165.188:38583 requested a full tablet report, sending...
02:03:36.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.391196 30358 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskxPorqF/build/debug/bin/kudu as pid 30775
02:03:36.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.391273 30358 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/wal/instance
02:03:36.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.391074 30903 heartbeater.cc:507] Master 127.29.165.190:39217 requested a full tablet report, sending...
02:03:36.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.392328 30400 ts_manager.cc:194] Registered new tserver with Master: 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130:35181)
02:03:36.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.392359 30544 ts_manager.cc:194] Registered new tserver with Master: 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130:35181)
02:03:36.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.393041 30544 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.165.130:57319
02:03:36.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.393934 30471 ts_manager.cc:194] Registered new tserver with Master: 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130:35181)
02:03:36.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.394780 30358 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskxPorqF/build/debug/bin/kudu
02:03:36.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskxPorqF/build/debug/bin/kudu
02:03:36.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/wal
02:03:36.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/data
02:03:36.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:03:36.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:03:36.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:03:36.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:03:36.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:03:36.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:03:36.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/logs
02:03:36.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/data/info.pb
02:03:36.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:03:36.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:03:36.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:03:36.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:03:36.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:03:36.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:03:36.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:03:36.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:03:36.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:03:36.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:03:36.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.165.131:0
02:03:36.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.165.131
02:03:36.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.165.131
02:03:36.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:03:36.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.165.190:39217,127.29.165.189:39733,127.29.165.188:38583
02:03:36.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.165.148:38003
02:03:36.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:03:36.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:03:36.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:03:36.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:36.493917 30910 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:03:36.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:36.494125 30910 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:03:36.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:36.494180 30910 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:03:36.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:36.497604 30910 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:03:36.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:36.497723 30910 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.165.131
02:03:36.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.502635 30910 tablet_server_runner.cc:78] Tablet server non-default flags:
02:03:36.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:03:36.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.165.148:38003
02:03:36.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:03:36.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:03:36.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/data
02:03:36.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/wal
02:03:36.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:03:36.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:03:36.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.165.131:0
02:03:36.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:03:36.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:03:36.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:03:36.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/data/info.pb
02:03:36.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.165.131
02:03:36.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:03:36.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.165.190:39217,127.29.165.189:39733,127.29.165.188:38583
02:03:36.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:03:36.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.30910
02:03:36.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:03:36.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:03:36.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:03:36.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:03:36.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.165.131
02:03:36.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/logs
02:03:36.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:03:36.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:03:36.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:03:36.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:03:36.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:03:36.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
02:03:36.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type DEBUG
02:03:36.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 01:43:15 UTC on bdcb31816ec0
02:03:36.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11948
02:03:36.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.503688 30910 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:03:36.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.504822 30910 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:03:36.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:36.511691 30915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:36.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:36.511696 30916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:36.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:36.511746 30918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:36.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.512131 30910 server_base.cc:1061] running on GCE node
02:03:36.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.512425 30910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:03:36.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.512866 30910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:03:36.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.514052 30910 hybrid_clock.cc:648] HybridClock initialized: now 1779069816514032 us; error 33 us; skew 500 ppm
02:03:36.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.515690 30910 webserver.cc:533] Webserver started at http://127.29.165.131:45365/ using document root <none> and password file <none>
02:03:36.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.516273 30910 fs_manager.cc:362] Metadata directory not provided
02:03:36.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.516352 30910 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:03:36.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.516584 30910 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
02:03:36.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.518004 30910 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/data/instance:
02:03:36.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "26ea15144933497a98a05f83b05e8f93"
02:03:36.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:03:36 on dist-test-slave-838s"
02:03:36.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.518551 30910 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/wal/instance:
02:03:36.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "26ea15144933497a98a05f83b05e8f93"
02:03:36.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:03:36 on dist-test-slave-838s"
02:03:36.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.521826 30910 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.002s	sys 0.001s
02:03:36.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.523993 30924 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:03:36.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.524950 30910 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
02:03:36.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.525074 30910 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/data,/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/wal
02:03:36.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "26ea15144933497a98a05f83b05e8f93"
02:03:36.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:03:36 on dist-test-slave-838s"
02:03:36.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.525158 30910 fs_report.cc:389] FS layout report
02:03:36.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:03:36.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/wal
02:03:36.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/wal
02:03:36.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/data/data
02:03:36.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:03:36.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:03:36.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:03:36.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:03:36.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:03:36.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:03:36.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:03:36.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:03:36.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:03:36.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:03:36.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:03:36.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.540134 30910 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:03:36.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.540742 30910 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:03:36.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.540918 30910 kserver.cc:163] Server-wide thread pool size limit: 3276
02:03:36.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.541421 30910 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:03:36.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.542346 30910 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
02:03:36.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.542411 30910 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
02:03:36.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.542479 30910 ts_tablet_manager.cc:616] Registered 0 tablets
02:03:36.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.542501 30910 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
02:03:36.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.551551 30910 rpc_server.cc:307] RPC server started. Bound to: 127.29.165.131:44535
02:03:36.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.551614 31037 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.165.131:44535 every 8 connection(s)
02:03:36.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.552531 30910 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/data/info.pb
02:03:36.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.559999 30358 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskxPorqF/build/debug/bin/kudu as pid 30910
02:03:36.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.560104 30358 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/wal/instance
02:03:36.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.560662 31040 heartbeater.cc:344] Connected to a master server at 127.29.165.189:39733
02:03:36.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.560842 31039 heartbeater.cc:344] Connected to a master server at 127.29.165.188:38583
02:03:36.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.560910 31039 heartbeater.cc:461] Registering TS with master...
02:03:36.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.560858 31040 heartbeater.cc:461] Registering TS with master...
02:03:36.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.561462 31039 heartbeater.cc:507] Master 127.29.165.188:38583 requested a full tablet report, sending...
02:03:36.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.561647 31040 heartbeater.cc:507] Master 127.29.165.189:39733 requested a full tablet report, sending...
02:03:36.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.562019 31038 heartbeater.cc:344] Connected to a master server at 127.29.165.190:39217
02:03:36.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.562100 31038 heartbeater.cc:461] Registering TS with master...
02:03:36.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.562652 30471 ts_manager.cc:194] Registered new tserver with Master: 26ea15144933497a98a05f83b05e8f93 (127.29.165.131:44535)
02:03:36.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.562652 30544 ts_manager.cc:194] Registered new tserver with Master: 26ea15144933497a98a05f83b05e8f93 (127.29.165.131:44535)
02:03:36.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.563431 31038 heartbeater.cc:507] Master 127.29.165.190:39217 requested a full tablet report, sending...
02:03:36.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.563388 30544 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.165.131:41407
02:03:36.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.564343 30400 ts_manager.cc:194] Registered new tserver with Master: 26ea15144933497a98a05f83b05e8f93 (127.29.165.131:44535)
02:03:36.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.566057 30358 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
02:03:36.567 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
02:03:36.568 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_masters {
}

02:03:36.576 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_masters {
  masters {
    id {
      type: MASTER
      index: 0
    }
    bound_rpc_address {
      host: "127.29.165.190"
      port: 39217
    }
    bound_http_address {
      host: "127.29.165.190"
      port: 35987
    }
  }
  masters {
    id {
      type: MASTER
      index: 1
    }
    bound_rpc_address {
      host: "127.29.165.189"
      port: 39733
    }
    bound_http_address {
      host: "127.29.165.189"
      port: 37455
    }
  }
  masters {
    id {
      type: MASTER
      index: 2
    }
    bound_rpc_address {
      host: "127.29.165.188"
      port: 38583
    }
    bound_http_address {
      host: "127.29.165.188"
      port: 45031
    }
  }
}

02:03:36.579 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_tservers {
}

02:03:36.581 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_tservers {
  tservers {
    id {
      type: TSERVER
      index: 0
    }
    bound_rpc_address {
      host: "127.29.165.129"
      port: 45737
    }
    bound_http_address {
      host: "127.29.165.129"
      port: 46663
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 1
    }
    bound_rpc_address {
      host: "127.29.165.130"
      port: 35181
    }
    bound_http_address {
      host: "127.29.165.130"
      port: 33955
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 2
    }
    bound_rpc_address {
      host: "127.29.165.131"
      port: 44535
    }
    bound_http_address {
      host: "127.29.165.131"
      port: 45365
    }
  }
}

02:03:36.581 [INFO - main] (KuduTestHarness.java:157) Creating a new Kudu client...
02:03:36.761 [DEBUG - main] (CompositeStrategy.java:59) Unable to load modern Subject methods, relying only on legacy methods
java.lang.NoSuchMethodException: javax.security.auth.Subject.current()
	at java.base/java.lang.Class.getDeclaredMethod(Class.java:2677)
	at org.apache.kudu.client.internals.ModernStrategy.<init>(ModernStrategy.java:51)
	at org.apache.kudu.client.internals.CompositeStrategy.<init>(CompositeStrategy.java:52)
	at org.apache.kudu.client.internals.CompositeStrategy.<clinit>(CompositeStrategy.java:41)
	at org.apache.kudu.client.internals.SecurityManagerCompatibility.get(SecurityManagerCompatibility.java:43)
	at org.apache.kudu.client.SecurityContext.setupSubject(SecurityContext.java:155)
	at org.apache.kudu.client.SecurityContext.<init>(SecurityContext.java:138)
	at org.apache.kudu.client.AsyncKuduClient.<init>(AsyncKuduClient.java:431)
	at org.apache.kudu.client.AsyncKuduClient$AsyncKuduClientBuilder.build(AsyncKuduClient.java:3109)
	at org.apache.kudu.test.KuduTestHarness.before(KuduTestHarness.java:161)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:50)
	at org.apache.kudu.test.junit.RetryRule$RetryStatement.doOneAttemptAndReport(RetryRule.java:184)
	at org.apache.kudu.test.junit.RetryRule$RetryStatement.evaluate(RetryRule.java:244)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:27)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
	at org.junit.runner.JUnitCore.runMain(JUnitCore.java:77)
	at org.junit.runner.JUnitCore.main(JUnitCore.java:36)
02:03:36.827 [DEBUG - main] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 
02:03:36.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.873704 30601 catalog_manager.cc:1294] Loaded cluster ID: f4a37041a44a4514a7b70d9da292b44a
02:03:36.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.873785 30601 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 324afe9044bf40299572cad6ecfc4d5d: loading cluster ID for follower catalog manager: success
02:03:36.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.876252 30601 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 324afe9044bf40299572cad6ecfc4d5d: acquiring CA information for follower catalog manager: success
02:03:36.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.877611 30601 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 324afe9044bf40299572cad6ecfc4d5d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
02:03:36.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.966068 30631 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 62a54aff7f4a46a9af3cc1f2d0e7d25b: acquiring CA information for follower catalog manager: success
02:03:36.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:36.967357 30631 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 62a54aff7f4a46a9af3cc1f2d0e7d25b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
02:03:37.012 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.29.165.189:39733(127.29.165.189:39733)] connecting to peer
02:03:37.070 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.29.165.188:38583(127.29.165.188:38583)] connecting to peer
02:03:37.074 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.29.165.190:39217(127.29.165.190:39217)] connecting to peer
02:03:37.086 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.29.165.188:38583(127.29.165.188:38583)] Successfully connected to peer
02:03:37.086 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.29.165.189:39733(127.29.165.189:39733)] Successfully connected to peer
02:03:37.086 [DEBUG - kudu-nio-2] (Connection.java:757) [peer master-127.29.165.190:39217(127.29.165.190:39217)] Successfully connected to peer
02:03:37.174 [DEBUG - kudu-nio-2] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.29.165.190
02:03:37.174 [DEBUG - kudu-nio-1] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.29.165.188
02:03:37.182 [DEBUG - kudu-nio-0] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.29.165.189
02:03:37.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.245177 30769 heartbeater.cc:499] Master 127.29.165.188:38583 was elected leader, sending a full tablet report...
02:03:37.381 [DEBUG - kudu-nio-2] (Negotiator.java:919) Authenticated connection [id: 0x9df57e10, L:/127.0.0.1:41380 - R:/127.29.165.190:39217] using SASL/PLAIN
02:03:37.381 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x6b6d5036, L:/127.0.0.1:57264 - R:/127.29.165.189:39733] using SASL/PLAIN
02:03:37.381 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0xb06a2230, L:/127.0.0.1:47968 - R:/127.29.165.188:38583] using SASL/PLAIN
02:03:37.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.394827 30904 heartbeater.cc:499] Master 127.29.165.188:38583 was elected leader, sending a full tablet report...
02:03:37.415 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet Kudu Master for table 'Kudu Master' with partition [<start>, <end>)
02:03:37.416 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=CreateTable, tablet=null, attempt=1, TimeoutTracker(timeout=50000, elapsed=446), Traces: [0ms] refreshing cache from master, [26ms] Sub RPC ConnectToMaster: sending RPC to server master-127.29.165.189:39733, [95ms] Sub RPC ConnectToMaster: sending RPC to server master-127.29.165.188:38583, [100ms] Sub RPC ConnectToMaster: sending RPC to server master-127.29.165.190:39217, [419ms] Sub RPC ConnectToMaster: received response from server master-127.29.165.190:39217: OK, [419ms] Sub RPC ConnectToMaster: received response from server master-127.29.165.189:39733: OK, [420ms] Sub RPC ConnectToMaster: received response from server master-127.29.165.188:38583: OK, deferred=null) after lookup
02:03:37.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.461921 30543 catalog_manager.cc:2282] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47968:
02:03:37.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) name: "org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956"
02:03:37.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) schema {
02:03:37.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
02:03:37.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "key"
02:03:37.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
02:03:37.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: true
02:03:37.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
02:03:37.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
02:03:37.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
02:03:37.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
02:03:37.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
02:03:37.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
02:03:37.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column1_i"
02:03:37.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
02:03:37.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
02:03:37.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
02:03:37.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
02:03:37.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
02:03:37.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
02:03:37.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
02:03:37.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
02:03:37.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column2_i"
02:03:37.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
02:03:37.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
02:03:37.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
02:03:37.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
02:03:37.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
02:03:37.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
02:03:37.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
02:03:37.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
02:03:37.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column3_s"
02:03:37.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: STRING
02:03:37.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
02:03:37.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: true
02:03:37.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     encoding: DICT_ENCODING
02:03:37.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     compression: LZ4
02:03:37.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 4096
02:03:37.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
02:03:37.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
02:03:37.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
02:03:37.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
02:03:37.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column4_b"
02:03:37.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: BOOL
02:03:37.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
02:03:37.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
02:03:37.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
02:03:37.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
02:03:37.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
02:03:37.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
02:03:37.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
02:03:37.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) partition_schema {
02:03:37.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   range_schema {
02:03:37.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     columns {
02:03:37.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)       name: "key"
02:03:37.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     }
02:03:37.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
02:03:37.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
02:03:37.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:37.463522 30543 catalog_manager.cc:7058] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
02:03:37.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.486119 30837 tablet_service.cc:1511] Processing CreateTablet for tablet 830b8d1af48f4d34b281a548f1dbe5ef (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956 [id=5cc77d019acb4678b65c30ec7f26e10b]), partition=RANGE (key) PARTITION UNBOUNDED
02:03:37.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.487186 30837 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 830b8d1af48f4d34b281a548f1dbe5ef. 1 dirs total, 0 dirs full, 0 dirs failed
02:03:37.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.491096 30702 tablet_service.cc:1511] Processing CreateTablet for tablet 830b8d1af48f4d34b281a548f1dbe5ef (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956 [id=5cc77d019acb4678b65c30ec7f26e10b]), partition=RANGE (key) PARTITION UNBOUNDED
02:03:37.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.492056 30702 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 830b8d1af48f4d34b281a548f1dbe5ef. 1 dirs total, 0 dirs full, 0 dirs failed
02:03:37.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.497644 31072 tablet_bootstrap.cc:492] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb: Bootstrap starting.
02:03:37.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.500500 31072 tablet_bootstrap.cc:654] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb: Neither blocks nor log segments found. Creating new log.
02:03:37.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.501505 31072 log.cc:826] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb: Log is configured to *not* fsync() on all Append() calls
02:03:37.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.501542 30972 tablet_service.cc:1511] Processing CreateTablet for tablet 830b8d1af48f4d34b281a548f1dbe5ef (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956 [id=5cc77d019acb4678b65c30ec7f26e10b]), partition=RANGE (key) PARTITION UNBOUNDED
02:03:37.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.502430 30972 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 830b8d1af48f4d34b281a548f1dbe5ef. 1 dirs total, 0 dirs full, 0 dirs failed
02:03:37.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.503413 31072 tablet_bootstrap.cc:492] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb: No bootstrap required, opened a new log
02:03:37.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.503604 31072 ts_tablet_manager.cc:1403] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb: Time spent bootstrapping tablet: real 0.006s	user 0.005s	sys 0.000s
02:03:37.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.506946 31072 raft_consensus.cc:359] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:03:37.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.507146 31072 raft_consensus.cc:385] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
02:03:37.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.507191 31072 raft_consensus.cc:740] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06f0647651a749f8a533afd451fe5fcb, State: Initialized, Role: FOLLOWER
02:03:37.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.507614 31072 consensus_queue.cc:260] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:03:37.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.508411 31072 ts_tablet_manager.cc:1434] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb: Time spent starting tablet: real 0.005s	user 0.001s	sys 0.003s
02:03:37.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.511744 31071 tablet_bootstrap.cc:492] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2: Bootstrap starting.
02:03:37.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.514070 31074 tablet_bootstrap.cc:492] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93: Bootstrap starting.
02:03:37.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.514616 31071 tablet_bootstrap.cc:654] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2: Neither blocks nor log segments found. Creating new log.
02:03:37.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.515648 31071 log.cc:826] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2: Log is configured to *not* fsync() on all Append() calls
02:03:37.517 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:894) Opened table 5cc77d019acb4678b65c30ec7f26e10b
02:03:37.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.519578 31074 tablet_bootstrap.cc:654] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93: Neither blocks nor log segments found. Creating new log.
02:03:37.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.520267 31071 tablet_bootstrap.cc:492] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2: No bootstrap required, opened a new log
02:03:37.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.520469 31071 ts_tablet_manager.cc:1403] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2: Time spent bootstrapping tablet: real 0.009s	user 0.004s	sys 0.000s
02:03:37.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.521085 31074 log.cc:826] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93: Log is configured to *not* fsync() on all Append() calls
02:03:37.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.523290 31074 tablet_bootstrap.cc:492] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93: No bootstrap required, opened a new log
02:03:37.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.523643 31074 ts_tablet_manager.cc:1403] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93: Time spent bootstrapping tablet: real 0.010s	user 0.005s	sys 0.000s
02:03:37.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.523742 31071 raft_consensus.cc:359] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:03:37.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.523947 31071 raft_consensus.cc:385] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
02:03:37.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.524011 31071 raft_consensus.cc:740] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e39ec54c2d642feafa81dc5d4f066c2, State: Initialized, Role: FOLLOWER
02:03:37.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.524479 31071 consensus_queue.cc:260] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:03:37.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.527590 31074 raft_consensus.cc:359] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:03:37.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.527751 31071 ts_tablet_manager.cc:1434] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
02:03:37.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.527805 31074 raft_consensus.cc:385] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
02:03:37.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.527858 31074 raft_consensus.cc:740] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26ea15144933497a98a05f83b05e8f93, State: Initialized, Role: FOLLOWER
02:03:37.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.528324 31074 consensus_queue.cc:260] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:03:37.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.529381 31074 ts_tablet_manager.cc:1434] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
02:03:37.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.530601 31039 heartbeater.cc:499] Master 127.29.165.188:38583 was elected leader, sending a full tablet report...
02:03:37.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.537097 31079 raft_consensus.cc:493] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
02:03:37.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.537235 31079 raft_consensus.cc:515] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:03:37.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.539628 31079 leader_election.cc:290] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130:35181), 06f0647651a749f8a533afd451fe5fcb (127.29.165.129:45737)
02:03:37.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.543033 30857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "830b8d1af48f4d34b281a548f1dbe5ef" candidate_uuid: "26ea15144933497a98a05f83b05e8f93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" is_pre_election: true
02:03:37.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.543233 30857 raft_consensus.cc:2468] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26ea15144933497a98a05f83b05e8f93 in term 0.
02:03:37.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.543627 30927 leader_election.cc:304] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26ea15144933497a98a05f83b05e8f93, 4e39ec54c2d642feafa81dc5d4f066c2; no voters: 
02:03:37.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.543941 30722 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "830b8d1af48f4d34b281a548f1dbe5ef" candidate_uuid: "26ea15144933497a98a05f83b05e8f93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f0647651a749f8a533afd451fe5fcb" is_pre_election: true
02:03:37.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.544169 30722 raft_consensus.cc:2468] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26ea15144933497a98a05f83b05e8f93 in term 0.
02:03:37.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.544431 31079 raft_consensus.cc:2804] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 0 FOLLOWER]: Leader pre-election won for term 1
02:03:37.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.544512 31079 raft_consensus.cc:493] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
02:03:37.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.544574 31079 raft_consensus.cc:3060] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 0 FOLLOWER]: Advancing to term 1
02:03:37.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.545383 31079 raft_consensus.cc:515] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:03:37.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.545791 31079 leader_election.cc:290] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [CANDIDATE]: Term 1 election: Requested vote from peers 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130:35181), 06f0647651a749f8a533afd451fe5fcb (127.29.165.129:45737)
02:03:37.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.546087 30857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "830b8d1af48f4d34b281a548f1dbe5ef" candidate_uuid: "26ea15144933497a98a05f83b05e8f93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e39ec54c2d642feafa81dc5d4f066c2"
02:03:37.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.546165 30722 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "830b8d1af48f4d34b281a548f1dbe5ef" candidate_uuid: "26ea15144933497a98a05f83b05e8f93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f0647651a749f8a533afd451fe5fcb"
02:03:37.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.546206 30857 raft_consensus.cc:3060] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 0 FOLLOWER]: Advancing to term 1
02:03:37.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.546277 30722 raft_consensus.cc:3060] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 0 FOLLOWER]: Advancing to term 1
02:03:37.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.547381 30857 raft_consensus.cc:2468] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26ea15144933497a98a05f83b05e8f93 in term 1.
02:03:37.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.547597 30722 raft_consensus.cc:2468] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26ea15144933497a98a05f83b05e8f93 in term 1.
02:03:37.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.547705 30927 leader_election.cc:304] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26ea15144933497a98a05f83b05e8f93, 4e39ec54c2d642feafa81dc5d4f066c2; no voters: 
02:03:37.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.547933 31079 raft_consensus.cc:2804] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 1 FOLLOWER]: Leader election won for term 1
02:03:37.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.548128 31079 raft_consensus.cc:697] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 1 LEADER]: Becoming Leader. State: Replica: 26ea15144933497a98a05f83b05e8f93, State: Running, Role: LEADER
02:03:37.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.548425 31079 consensus_queue.cc:237] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:03:37.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.551584 30543 catalog_manager.cc:5696] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26ea15144933497a98a05f83b05e8f93 (127.29.165.131). New cstate: current_term: 1 leader_uuid: "26ea15144933497a98a05f83b05e8f93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } health_report { overall_health: HEALTHY } } }
02:03:37.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:37.554788 31041 tablet.cc:2404] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93: Can't schedule compaction. Clean time has not been advanced past its initial value.
02:03:37.587 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:894) Opened table 5cc77d019acb4678b65c30ec7f26e10b
02:03:37.605 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:03:37.613 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299992, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737),26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)[L],4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)]}]
02:03:37.614 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=null, attempt=1, sequence_id=-1, TimeoutTracker(timeout=30000, elapsed=18), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [7ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=null) after lookup
02:03:37.623 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] connecting to peer
02:03:37.625 [DEBUG - kudu-nio-3] (Connection.java:757) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] Successfully connected to peer
02:03:37.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:37.633949 30906 tablet.cc:2404] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
02:03:37.690 [DEBUG - kudu-nio-3] (Negotiator.java:919) Authenticated connection [id: 0x6400334a, L:/127.0.0.1:50388 - R:/127.29.165.131:44535] using TOKEN/null
02:03:37.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.698494 30722 raft_consensus.cc:1275] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 1 FOLLOWER]: Refusing update from remote peer 26ea15144933497a98a05f83b05e8f93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
02:03:37.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.699203 30857 raft_consensus.cc:1275] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 1 FOLLOWER]: Refusing update from remote peer 26ea15144933497a98a05f83b05e8f93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
02:03:37.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.699926 31079 consensus_queue.cc:1048] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
02:03:37.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.700296 31082 consensus_queue.cc:1048] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
02:03:37.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.712724 31085 mvcc.cc:204] Tried to move back new op lower bound from 7287069973285478400 to 7287069972681011200. Current Snapshot: MvccSnapshot[applied={T|T < 7287069973285478400}]
02:03:37.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.715221 31088 mvcc.cc:204] Tried to move back new op lower bound from 7287069973285478400 to 7287069972681011200. Current Snapshot: MvccSnapshot[applied={T|T < 7287069973285478400}]
02:03:37.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.715797 31086 mvcc.cc:204] Tried to move back new op lower bound from 7287069973285478400 to 7287069972681011200. Current Snapshot: MvccSnapshot[applied={T|T < 7287069973285478400 or (T in {7287069973285478400})}]
02:03:37.793 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 3 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956, tablet=830b8d1af48f4d34b281a548f1dbe5ef, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:03:37.796 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.29.165.131:44535
02:03:37.797 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

02:03:37.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:37.798190 30358 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskxPorqF/build/debug/bin/kudu with pid 30910
02:03:37.807 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:03:37.811 [DEBUG - kudu-nio-3] (Connection.java:239) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] handling channelInactive
02:03:37.812 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] cleaning up while in state READY due to: connection closed
02:03:37.812 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: connection closed
02:03:37.835 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:03:37.835 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737),26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)[L],4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)]}]
02:03:37.836 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=2, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=28), Traces: [0ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [5ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: connection closed, [5ms] delaying RPC due to: Network error: connection closed, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [28ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@433514257(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@2c818bf9 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1b3e7a21 -> wakeup thread Time-limited test)) after lookup
02:03:37.836 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] connecting to peer
02:03:37.840 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:37.840 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] cleaning up while in state CONNECTING due to: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:37.840 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:37.854 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:03:37.854 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737),26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)[L],4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)]}]
02:03:37.854 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=4, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=47), Traces: [0ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [5ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: connection closed, [5ms] delaying RPC due to: Network error: connection closed, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [28ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [29ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [33ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [34ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [44ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [46ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@433514257(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@2c818bf9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1470856632 after retry RPC@1903160225), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1b3e7a21 -> wakeup thread Time-limited test -> (continuation of Deferred@1470856632 after retry RPC@1903160225))) after lookup
02:03:37.855 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] connecting to peer
02:03:37.857 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:37.857 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] cleaning up while in state CONNECTING due to: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:37.857 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:37.895 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:03:37.896 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737),26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)[L],4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)]}]
02:03:37.896 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=6, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=89), Traces: [0ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [5ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: connection closed, [5ms] delaying RPC due to: Network error: connection closed, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [28ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [29ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [33ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [34ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [44ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [46ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [48ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [50ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [50ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [84ms] refreshing cache from master, [85ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [88ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@433514257(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@2c818bf9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1470856632 after retry RPC@1903160225) -> (continuation of Deferred@667013939 after retry RPC@96380588), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1b3e7a21 -> wakeup thread Time-limited test -> (continuation of Deferred@1470856632 after retry RPC@1903160225) -> (continuation of Deferred@667013939 after retry RPC@96380588))) after lookup
02:03:37.897 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] connecting to peer
02:03:37.900 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:37.900 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] cleaning up while in state CONNECTING due to: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:37.900 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:38.015 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:03:38.015 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737),26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)[L],4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)]}]
02:03:38.015 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=8, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=208), Traces: [0ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [5ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: connection closed, [5ms] delaying RPC due to: Network error: connection closed, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [28ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [29ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [33ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [34ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [44ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [46ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [48ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [50ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [50ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [84ms] refreshing cache from master, [85ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [88ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [90ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [93ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [93ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [204ms] refreshing cache from master, [204ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [207ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@433514257(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@2c818bf9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1470856632 after retry RPC@1903160225) -> (continuation of Deferred@667013939 after retry RPC@96380588) -> (continuation of Deferred@123256457 after retry RPC@981643234), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1b3e7a21 -> wakeup thread Time-limited test -> (continuation of Deferred@1470856632 after retry RPC@1903160225) -> (continuation of Deferred@667013939 after retry RPC@96380588) -> (continuation of Deferred@123256457 after retry RPC@981643234))) after lookup
02:03:38.016 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] connecting to peer
02:03:38.018 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:38.018 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] cleaning up while in state CONNECTING due to: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:38.018 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:38.514 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:03:38.514 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737),26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)[L],4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)]}]
02:03:38.515 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=10, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=707), Traces: [0ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [5ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: connection closed, [5ms] delaying RPC due to: Network error: connection closed, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [28ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [29ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [33ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [34ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [44ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [46ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [48ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [50ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [50ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [84ms] refreshing cache from master, [85ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [88ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [90ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [93ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [93ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [204ms] refreshing cache from master, [204ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [207ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [209ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [211ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [211ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [704ms] refreshing cache from master, [704ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [707ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@433514257(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@2c818bf9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1470856632 after retry RPC@1903160225) -> (continuation of Deferred@667013939 after retry RPC@96380588) -> (continuation of Deferred@123256457 after retry RPC@981643234) -> (continuation of Deferred@1982532097 after retry RPC@622733508), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1b3e7a21 -> wakeup thread Time-limited test -> (continuation of Deferred@1470856632 after retry RPC@1903160225) -> (continuation of Deferred@667013939 after retry RPC@96380588) -> (continuation of Deferred@123256457 after retry RPC@981643234) -> (continuation of Deferred@1982532097 after retry RPC@622733508))) after lookup
02:03:38.516 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] connecting to peer
02:03:38.517 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:38.517 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] cleaning up while in state CONNECTING due to: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:38.517 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:38.674 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:03:38.675 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737),26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)[L],4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)]}]
02:03:38.675 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=12, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=868), Traces: [0ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [5ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: connection closed, [5ms] delaying RPC due to: Network error: connection closed, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [28ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [29ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [33ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [34ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [44ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [46ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [48ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [50ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [50ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [84ms] refreshing cache from master, [85ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [88ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [90ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [93ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [93ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [204ms] refreshing cache from master, [204ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [207ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [209ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [211ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [211ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [704ms] refreshing cache from master, [704ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [707ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [708ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [710ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [710ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [864ms] refreshing cache from master, [864ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [867ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@433514257(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@2c818bf9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1470856632 after retry RPC@1903160225) -> (continuation of Deferred@667013939 after retry RPC@96380588) -> (continuation of Deferred@123256457 after retry RPC@981643234) -> (continuation of Deferred@1982532097 after retry RPC@622733508) -> (continuation of Deferred@1004355622 after retry RPC@1867581329), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1b3e7a21 -> wakeup thread Time-limited test -> (continuation of Deferred@1470856632 after retry RPC@1903160225) -> (continuation of Deferred@667013939 after retry RPC@96380588) -> (continuation of Deferred@123256457 after retry RPC@981643234) -> (continuation of Deferred@1982532097 after retry RPC@622733508) -> (continuation of Deferred@1004355622 after retry RPC@1867581329))) after lookup
02:03:38.676 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] connecting to peer
02:03:38.677 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:38.677 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] cleaning up while in state CONNECTING due to: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:38.677 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:39.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:39.250978 31098 raft_consensus.cc:493] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26ea15144933497a98a05f83b05e8f93)
02:03:39.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:39.251150 31098 raft_consensus.cc:515] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:03:39.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:39.251818 31099 raft_consensus.cc:493] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26ea15144933497a98a05f83b05e8f93)
02:03:39.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:39.251940 31099 raft_consensus.cc:515] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:03:39.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:39.252449 31098 leader_election.cc:290] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130:35181), 26ea15144933497a98a05f83b05e8f93 (127.29.165.131:44535)
02:03:39.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:39.253326 30657 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.165.131:44535: connect: Connection refused (error 111)
02:03:39.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:39.253492 31099 leader_election.cc:290] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 06f0647651a749f8a533afd451fe5fcb (127.29.165.129:45737), 26ea15144933497a98a05f83b05e8f93 (127.29.165.131:44535)
02:03:39.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:39.254401 30792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.165.131:44535: connect: Connection refused (error 111)
02:03:39.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:39.254864 30657 leader_election.cc:336] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 26ea15144933497a98a05f83b05e8f93 (127.29.165.131:44535): Network error: Client connection negotiation failed: client connection to 127.29.165.131:44535: connect: Connection refused (error 111)
02:03:39.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:39.255407 30792 leader_election.cc:336] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 26ea15144933497a98a05f83b05e8f93 (127.29.165.131:44535): Network error: Client connection negotiation failed: client connection to 127.29.165.131:44535: connect: Connection refused (error 111)
02:03:39.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:39.256731 30857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "830b8d1af48f4d34b281a548f1dbe5ef" candidate_uuid: "06f0647651a749f8a533afd451fe5fcb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" is_pre_election: true
02:03:39.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:39.256875 30857 raft_consensus.cc:2468] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06f0647651a749f8a533afd451fe5fcb in term 1.
02:03:39.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:39.257185 30657 leader_election.cc:304] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06f0647651a749f8a533afd451fe5fcb, 4e39ec54c2d642feafa81dc5d4f066c2; no voters: 26ea15144933497a98a05f83b05e8f93
02:03:39.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:39.257416 31098 raft_consensus.cc:2804] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 1 FOLLOWER]: Leader pre-election won for term 2
02:03:39.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:39.257450 30722 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "830b8d1af48f4d34b281a548f1dbe5ef" candidate_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "06f0647651a749f8a533afd451fe5fcb" is_pre_election: true
02:03:39.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:39.257494 31098 raft_consensus.cc:493] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 1 FOLLOWER]: Starting leader election (detected failure of leader 26ea15144933497a98a05f83b05e8f93)
02:03:39.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:39.257534 31098 raft_consensus.cc:3060] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 1 FOLLOWER]: Advancing to term 2
02:03:39.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:39.258279 31098 raft_consensus.cc:515] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:03:39.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:39.258527 30722 raft_consensus.cc:2393] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e39ec54c2d642feafa81dc5d4f066c2 in current term 2: Already voted for candidate 06f0647651a749f8a533afd451fe5fcb in this term.
02:03:39.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:39.258671 31098 leader_election.cc:290] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [CANDIDATE]: Term 2 election: Requested vote from peers 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130:35181), 26ea15144933497a98a05f83b05e8f93 (127.29.165.131:44535)
02:03:39.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:39.259168 30793 leader_election.cc:304] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e39ec54c2d642feafa81dc5d4f066c2; no voters: 06f0647651a749f8a533afd451fe5fcb, 26ea15144933497a98a05f83b05e8f93
02:03:39.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:39.259300 30857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "830b8d1af48f4d34b281a548f1dbe5ef" candidate_uuid: "06f0647651a749f8a533afd451fe5fcb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e39ec54c2d642feafa81dc5d4f066c2"
02:03:39.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:39.259387 30857 raft_consensus.cc:3060] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 1 FOLLOWER]: Advancing to term 2
02:03:39.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:39.259843 30657 leader_election.cc:336] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 26ea15144933497a98a05f83b05e8f93 (127.29.165.131:44535): Network error: Client connection negotiation failed: client connection to 127.29.165.131:44535: connect: Connection refused (error 111)
02:03:39.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:39.260335 30857 raft_consensus.cc:2468] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06f0647651a749f8a533afd451fe5fcb in term 2.
02:03:39.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:39.260478 31099 raft_consensus.cc:2749] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
02:03:39.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:39.260689 30657 leader_election.cc:304] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06f0647651a749f8a533afd451fe5fcb, 4e39ec54c2d642feafa81dc5d4f066c2; no voters: 26ea15144933497a98a05f83b05e8f93
02:03:39.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:39.260903 31098 raft_consensus.cc:2804] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 2 FOLLOWER]: Leader election won for term 2
02:03:39.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:39.261094 31098 raft_consensus.cc:697] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 2 LEADER]: Becoming Leader. State: Replica: 06f0647651a749f8a533afd451fe5fcb, State: Running, Role: LEADER
02:03:39.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:39.261379 31098 consensus_queue.cc:237] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:03:39.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:39.264649 30543 catalog_manager.cc:5696] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb reported cstate change: term changed from 1 to 2, leader changed from 26ea15144933497a98a05f83b05e8f93 (127.29.165.131) to 06f0647651a749f8a533afd451fe5fcb (127.29.165.129). New cstate: current_term: 2 leader_uuid: "06f0647651a749f8a533afd451fe5fcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } health_report { overall_health: UNKNOWN } } }
02:03:39.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:39.645429 30657 consensus_peers.cc:597] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb -> Peer 26ea15144933497a98a05f83b05e8f93 (127.29.165.131:44535): Couldn't send request to peer 26ea15144933497a98a05f83b05e8f93. Status: Network error: Client connection negotiation failed: client connection to 127.29.165.131:44535: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:03:39.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:39.867866 30857 raft_consensus.cc:1275] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 2 FOLLOWER]: Refusing update from remote peer 06f0647651a749f8a533afd451fe5fcb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
02:03:39.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:39.868469 31098 consensus_queue.cc:1048] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
02:03:41.794 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:03:41.795 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)[L],26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535),4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)]}]
02:03:41.795 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=14, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=3988), Traces: [0ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [5ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: connection closed, [5ms] delaying RPC due to: Network error: connection closed, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [28ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [29ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [33ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [34ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [44ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [46ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [48ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [50ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [50ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [84ms] refreshing cache from master, [85ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [88ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [90ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [93ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [93ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [204ms] refreshing cache from master, [204ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [207ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [209ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [211ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [211ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [704ms] refreshing cache from master, [704ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [707ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [708ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [710ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [710ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [864ms] refreshing cache from master, [864ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [867ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [869ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [870ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [870ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [3984ms] refreshing cache from master, [3984ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [3987ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@433514257(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@2c818bf9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1470856632 after retry RPC@1903160225) -> (continuation of Deferred@667013939 after retry RPC@96380588) -> (continuation of Deferred@123256457 after retry RPC@981643234) -> (continuation of Deferred@1982532097 after retry RPC@622733508) -> (continuation of Deferred@1004355622 after retry RPC@1867581329) -> (continuation of Deferred@1769580124 after retry RPC@293437161), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1b3e7a21 -> wakeup thread Time-limited test -> (continuation of Deferred@1470856632 after retry RPC@1903160225) -> (continuation of Deferred@667013939 after retry RPC@96380588) -> (continuation of Deferred@123256457 after retry RPC@981643234) -> (continuation of Deferred@1982532097 after retry RPC@622733508) -> (continuation of Deferred@1004355622 after retry RPC@1867581329) -> (continuation of Deferred@1769580124 after retry RPC@293437161))) after lookup
02:03:41.796 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] connecting to peer
02:03:41.797 [DEBUG - kudu-nio-2] (Connection.java:757) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] Successfully connected to peer
02:03:41.818 [DEBUG - kudu-nio-2] (Negotiator.java:919) Authenticated connection [id: 0x39965650, L:/127.0.0.1:39588 - R:/127.29.165.129:45737] using TOKEN/null
02:03:41.843 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.29.165.131:44535
02:03:41.845 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

02:03:41.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:41.845777 30358 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskxPorqF/build/debug/bin/kudu
02:03:41.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskxPorqF/build/debug/bin/kudu
02:03:41.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/wal
02:03:41.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/data
02:03:41.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:03:41.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:03:41.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:03:41.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:03:41.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:03:41.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:03:41.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/logs
02:03:41.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/data/info.pb
02:03:41.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:03:41.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:03:41.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:03:41.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:03:41.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:03:41.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:03:41.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:03:41.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:03:41.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:03:41.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:03:41.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.165.131:44535
02:03:41.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.165.131
02:03:41.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.165.190:39217,127.29.165.189:39733,127.29.165.188:38583
02:03:41.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45365
02:03:41.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.165.131
02:03:41.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.165.148:38003
02:03:41.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:03:41.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:03:41.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:03:41.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:41.952337 31138 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:03:41.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:41.952569 31138 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:03:41.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:41.952636 31138 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:03:41.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:41.956168 31138 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:03:41.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:41.956319 31138 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.165.131
02:03:41.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:41.960213 31138 tablet_server_runner.cc:78] Tablet server non-default flags:
02:03:41.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:03:41.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.165.148:38003
02:03:41.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:03:41.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:03:41.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/data
02:03:41.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/wal
02:03:41.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:03:41.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:03:41.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.165.131:44535
02:03:41.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:03:41.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:03:41.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:03:41.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/data/info.pb
02:03:41.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.165.131
02:03:41.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45365
02:03:41.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.165.190:39217,127.29.165.189:39733,127.29.165.188:38583
02:03:41.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:03:41.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.31138
02:03:41.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:03:41.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:03:41.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:03:41.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:03:41.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.165.131
02:03:41.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/logs
02:03:41.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:03:41.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:03:41.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:03:41.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:03:41.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:03:41.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
02:03:41.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type DEBUG
02:03:41.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 01:43:15 UTC on bdcb31816ec0
02:03:41.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11948
02:03:41.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:41.961575 31138 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:03:41.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:41.962904 31138 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:03:41.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:41.969478 31146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:41.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:41.969488 31144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:41.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:41.969544 31138 server_base.cc:1061] running on GCE node
02:03:41.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:41.969487 31143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:41.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:41.970153 31138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:03:41.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:41.970654 31138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:03:41.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:41.971861 31138 hybrid_clock.cc:648] HybridClock initialized: now 1779069821971851 us; error 30 us; skew 500 ppm
02:03:41.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:41.973712 31138 webserver.cc:533] Webserver started at http://127.29.165.131:45365/ using document root <none> and password file <none>
02:03:41.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:41.974442 31138 fs_manager.cc:362] Metadata directory not provided
02:03:41.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:41.974545 31138 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:03:41.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:41.977875 31138 fs_manager.cc:714] Time spent opening directory manager: real 0.002s	user 0.002s	sys 0.000s
02:03:41.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:41.979722 31152 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:03:41.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:41.980641 31138 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
02:03:41.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:41.980764 31138 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/data,/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/wal
02:03:41.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "26ea15144933497a98a05f83b05e8f93"
02:03:41.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:03:36 on dist-test-slave-838s"
02:03:41.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:41.981176 31138 fs_report.cc:389] FS layout report
02:03:41.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:03:41.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/wal
02:03:41.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/wal
02:03:41.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/data/data
02:03:41.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:03:41.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:03:41.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:03:41.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:03:41.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:03:41.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:03:41.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:03:41.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:03:41.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:03:41.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:03:41.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:03:41.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:41.995177 31138 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:03:41.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:41.995885 31138 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:03:41.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:41.996083 31138 kserver.cc:163] Server-wide thread pool size limit: 3276
02:03:41.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:41.996644 31138 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:03:41.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:41.997932 31159 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
02:03:42.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:42.001199 31138 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:03:42.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:42.001276 31138 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.004s	user 0.001s	sys 0.000s
02:03:42.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:42.001336 31138 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:03:42.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:42.002866 31138 ts_tablet_manager.cc:616] Registered 1 tablets
02:03:42.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:42.002921 31138 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s	user 0.002s	sys 0.000s
02:03:42.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:42.003221 31159 tablet_bootstrap.cc:492] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93: Bootstrap starting.
02:03:42.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:42.015101 31266 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.165.131:44535 every 8 connection(s)
02:03:42.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:42.015074 31138 rpc_server.cc:307] RPC server started. Bound to: 127.29.165.131:44535
02:03:42.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:42.016649 31138 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/data/info.pb
02:03:42.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:42.020967 31159 log.cc:826] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93: Log is configured to *not* fsync() on all Append() calls
02:03:42.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:42.022215 30358 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskxPorqF/build/debug/bin/kudu as pid 31138
02:03:42.023 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:03:42.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:42.029335 31268 heartbeater.cc:344] Connected to a master server at 127.29.165.188:38583
02:03:42.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:42.029589 31268 heartbeater.cc:461] Registering TS with master...
02:03:42.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:42.029632 31269 heartbeater.cc:344] Connected to a master server at 127.29.165.189:39733
02:03:42.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:42.029716 31269 heartbeater.cc:461] Registering TS with master...
02:03:42.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:42.030167 31269 heartbeater.cc:507] Master 127.29.165.189:39733 requested a full tablet report, sending...
02:03:42.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:42.031731 31159 tablet_bootstrap.cc:492] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:03:42.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:42.031816 31267 heartbeater.cc:344] Connected to a master server at 127.29.165.190:39217
02:03:42.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:42.032069 31267 heartbeater.cc:461] Registering TS with master...
02:03:42.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:42.032263 31267 heartbeater.cc:507] Master 127.29.165.190:39217 requested a full tablet report, sending...
02:03:42.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:42.032333 31268 heartbeater.cc:507] Master 127.29.165.188:38583 requested a full tablet report, sending...
02:03:42.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:42.032369 31159 tablet_bootstrap.cc:492] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93: Bootstrap complete.
02:03:42.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:42.033427 30400 ts_manager.cc:194] Re-registered known tserver with Master: 26ea15144933497a98a05f83b05e8f93 (127.29.165.131:44535)
02:03:42.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:42.033479 30471 ts_manager.cc:194] Re-registered known tserver with Master: 26ea15144933497a98a05f83b05e8f93 (127.29.165.131:44535)
02:03:42.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:42.033635 31159 ts_tablet_manager.cc:1403] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93: Time spent bootstrapping tablet: real 0.031s	user 0.016s	sys 0.011s
02:03:42.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:42.034096 30543 ts_manager.cc:194] Re-registered known tserver with Master: 26ea15144933497a98a05f83b05e8f93 (127.29.165.131:44535)
02:03:42.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:42.035043 30543 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.165.131:58249
02:03:42.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:42.036259 31159 raft_consensus.cc:359] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:03:42.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:42.036790 31159 raft_consensus.cc:740] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26ea15144933497a98a05f83b05e8f93, State: Initialized, Role: FOLLOWER
02:03:42.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:42.037361 31159 consensus_queue.cc:260] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:03:42.037 [DEBUG - kudu-nio-2] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 6 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956, tablet=830b8d1af48f4d34b281a548f1dbe5ef, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:03:42.038 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.29.165.129:45737
02:03:42.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:42.038401 31159 ts_tablet_manager.cc:1434] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.000s
02:03:42.038 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

02:03:42.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:42.039026 30358 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskxPorqF/build/debug/bin/kudu with pid 30640
02:03:42.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:42.041074 31268 heartbeater.cc:499] Master 127.29.165.188:38583 was elected leader, sending a full tablet report...
02:03:42.046 [DEBUG - kudu-nio-2] (Connection.java:239) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] handling channelInactive
02:03:42.046 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:03:42.046 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] cleaning up while in state READY due to: connection closed
02:03:42.047 [DEBUG - Time-limited test] (Connection.java:748) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] connecting to peer
02:03:42.048 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:03:42.049 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] cleaning up while in state CONNECTING due to: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:03:42.049 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:03:42.054 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:03:42.055 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)[L],26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535),4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)]}]
02:03:42.055 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=2, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=8), Traces: [0ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [2ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [2ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [7ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@85502774(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@7f68074a -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@74968804 -> wakeup thread Time-limited test)) after lookup
02:03:42.056 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] connecting to peer
02:03:42.057 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:03:42.057 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] cleaning up while in state CONNECTING due to: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:03:42.058 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:03:42.074 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:03:42.075 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)[L],26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535),4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)]}]
02:03:42.075 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=4, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=28), Traces: [0ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [2ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [2ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [7ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [9ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [11ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [11ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [27ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@85502774(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@7f68074a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@385979398 after retry RPC@1461184586), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@74968804 -> wakeup thread Time-limited test -> (continuation of Deferred@385979398 after retry RPC@1461184586))) after lookup
02:03:42.076 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] connecting to peer
02:03:42.077 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:03:42.077 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] cleaning up while in state CONNECTING due to: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:03:42.077 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:03:42.114 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:03:42.114 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)[L],26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535),4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)]}]
02:03:42.115 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=6, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=68), Traces: [0ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [2ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [2ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [7ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [9ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [11ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [11ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [27ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [28ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [30ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [30ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [67ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@85502774(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@7f68074a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@385979398 after retry RPC@1461184586) -> (continuation of Deferred@106893805 after retry RPC@1851976916), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@74968804 -> wakeup thread Time-limited test -> (continuation of Deferred@385979398 after retry RPC@1461184586) -> (continuation of Deferred@106893805 after retry RPC@1851976916))) after lookup
02:03:42.116 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] connecting to peer
02:03:42.117 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:03:42.117 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] cleaning up while in state CONNECTING due to: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:03:42.117 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:03:42.154 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:03:42.155 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)[L],26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535),4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)]}]
02:03:42.156 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=8, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=108), Traces: [0ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [2ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [2ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [7ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [9ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [11ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [11ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [27ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [28ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [30ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [30ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [67ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [68ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [70ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [71ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [104ms] refreshing cache from master, [105ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [107ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@85502774(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@7f68074a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@385979398 after retry RPC@1461184586) -> (continuation of Deferred@106893805 after retry RPC@1851976916) -> (continuation of Deferred@1536646205 after retry RPC@577376215), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@74968804 -> wakeup thread Time-limited test -> (continuation of Deferred@385979398 after retry RPC@1461184586) -> (continuation of Deferred@106893805 after retry RPC@1851976916) -> (continuation of Deferred@1536646205 after retry RPC@577376215))) after lookup
02:03:42.156 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] connecting to peer
02:03:42.158 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:03:42.158 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] cleaning up while in state CONNECTING due to: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:03:42.158 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:03:42.454 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:03:42.454 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)[L],26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535),4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)]}]
02:03:42.455 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=10, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=407), Traces: [0ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [2ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [2ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [7ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [9ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [11ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [11ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [27ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [28ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [30ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [30ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [67ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [68ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [70ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [71ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [104ms] refreshing cache from master, [105ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [107ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [109ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [111ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [111ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [404ms] refreshing cache from master, [404ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [407ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@85502774(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@7f68074a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@385979398 after retry RPC@1461184586) -> (continuation of Deferred@106893805 after retry RPC@1851976916) -> (continuation of Deferred@1536646205 after retry RPC@577376215) -> (continuation of Deferred@767133785 after retry RPC@1240430424), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@74968804 -> wakeup thread Time-limited test -> (continuation of Deferred@385979398 after retry RPC@1461184586) -> (continuation of Deferred@106893805 after retry RPC@1851976916) -> (continuation of Deferred@1536646205 after retry RPC@577376215) -> (continuation of Deferred@767133785 after retry RPC@1240430424))) after lookup
02:03:42.456 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] connecting to peer
02:03:42.457 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:03:42.457 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] cleaning up while in state CONNECTING due to: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:03:42.457 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:03:43.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:43.290422 31280 raft_consensus.cc:493] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
02:03:43.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:43.290658 31280 raft_consensus.cc:515] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:03:43.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:43.291846 31280 leader_election.cc:290] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130:35181), 06f0647651a749f8a533afd451fe5fcb (127.29.165.129:45737)
02:03:43.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:43.292656 31156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.165.129:45737: connect: Connection refused (error 111)
02:03:43.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:43.293458 31156 leader_election.cc:336] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 06f0647651a749f8a533afd451fe5fcb (127.29.165.129:45737): Network error: Client connection negotiation failed: client connection to 127.29.165.129:45737: connect: Connection refused (error 111)
02:03:43.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:43.295928 30857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "830b8d1af48f4d34b281a548f1dbe5ef" candidate_uuid: "26ea15144933497a98a05f83b05e8f93" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" is_pre_election: true
02:03:43.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:43.296442 31155 leader_election.cc:304] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26ea15144933497a98a05f83b05e8f93; no voters: 06f0647651a749f8a533afd451fe5fcb, 4e39ec54c2d642feafa81dc5d4f066c2
02:03:43.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:43.296725 31280 raft_consensus.cc:2749] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
02:03:43.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:43.342548 31285 raft_consensus.cc:493] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 06f0647651a749f8a533afd451fe5fcb)
02:03:43.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:43.342741 31285 raft_consensus.cc:515] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:03:43.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:43.343297 31285 leader_election.cc:290] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 06f0647651a749f8a533afd451fe5fcb (127.29.165.129:45737), 26ea15144933497a98a05f83b05e8f93 (127.29.165.131:44535)
02:03:43.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:43.344571 30793 leader_election.cc:336] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 06f0647651a749f8a533afd451fe5fcb (127.29.165.129:45737): Network error: Client connection negotiation failed: client connection to 127.29.165.129:45737: connect: Connection refused (error 111)
02:03:43.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:43.347757 31221 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "830b8d1af48f4d34b281a548f1dbe5ef" candidate_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "26ea15144933497a98a05f83b05e8f93" is_pre_election: true
02:03:43.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:43.347954 31221 raft_consensus.cc:2468] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e39ec54c2d642feafa81dc5d4f066c2 in term 1.
02:03:43.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:43.348573 30792 leader_election.cc:304] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26ea15144933497a98a05f83b05e8f93, 4e39ec54c2d642feafa81dc5d4f066c2; no voters: 06f0647651a749f8a533afd451fe5fcb
02:03:43.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:43.348747 31285 raft_consensus.cc:2804] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 2 FOLLOWER]: Leader pre-election won for term 3
02:03:43.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:43.348814 31285 raft_consensus.cc:493] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 06f0647651a749f8a533afd451fe5fcb)
02:03:43.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:43.348855 31285 raft_consensus.cc:3060] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 2 FOLLOWER]: Advancing to term 3
02:03:43.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:43.349712 31285 raft_consensus.cc:515] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:03:43.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:43.350088 31285 leader_election.cc:290] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [CANDIDATE]: Term 3 election: Requested vote from peers 06f0647651a749f8a533afd451fe5fcb (127.29.165.129:45737), 26ea15144933497a98a05f83b05e8f93 (127.29.165.131:44535)
02:03:43.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:43.350428 31221 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "830b8d1af48f4d34b281a548f1dbe5ef" candidate_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "26ea15144933497a98a05f83b05e8f93"
02:03:43.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:43.350554 31221 raft_consensus.cc:3060] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 1 FOLLOWER]: Advancing to term 3
02:03:43.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:43.351092 30793 leader_election.cc:336] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 06f0647651a749f8a533afd451fe5fcb (127.29.165.129:45737): Network error: Client connection negotiation failed: client connection to 127.29.165.129:45737: connect: Connection refused (error 111)
02:03:43.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:43.352150 31221 raft_consensus.cc:2468] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e39ec54c2d642feafa81dc5d4f066c2 in term 3.
02:03:43.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:43.352438 30792 leader_election.cc:304] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26ea15144933497a98a05f83b05e8f93, 4e39ec54c2d642feafa81dc5d4f066c2; no voters: 06f0647651a749f8a533afd451fe5fcb
02:03:43.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:43.352619 31285 raft_consensus.cc:2804] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 3 FOLLOWER]: Leader election won for term 3
02:03:43.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:43.352779 31285 raft_consensus.cc:697] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 3 LEADER]: Becoming Leader. State: Replica: 4e39ec54c2d642feafa81dc5d4f066c2, State: Running, Role: LEADER
02:03:43.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:43.353068 31285 consensus_queue.cc:237] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:03:43.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:43.355234 30543 catalog_manager.cc:5696] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 reported cstate change: term changed from 2 to 3, leader changed from 06f0647651a749f8a533afd451fe5fcb (127.29.165.129) to 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130). New cstate: current_term: 3 leader_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } health_report { overall_health: UNKNOWN } } }
02:03:43.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:43.736100 31221 raft_consensus.cc:1275] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 3 FOLLOWER]: Refusing update from remote peer 4e39ec54c2d642feafa81dc5d4f066c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
02:03:43.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:43.736863 31285 consensus_queue.cc:1048] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 4, Time since last communication: 0.000s
02:03:43.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:43.748236 30793 consensus_peers.cc:597] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 -> Peer 06f0647651a749f8a533afd451fe5fcb (127.29.165.129:45737): Couldn't send request to peer 06f0647651a749f8a533afd451fe5fcb. Status: Network error: Client connection negotiation failed: client connection to 127.29.165.129:45737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:03:43.854 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:03:43.854 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737),26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535),4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)[L]]}]
02:03:43.855 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=12, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=1807), Traces: [0ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [2ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [2ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [7ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [9ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [11ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [11ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [27ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [28ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [30ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [30ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [64ms] refreshing cache from master, [64ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [67ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [68ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [70ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [71ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [104ms] refreshing cache from master, [105ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [107ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [109ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [111ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [111ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [404ms] refreshing cache from master, [404ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [407ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [408ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [410ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [410ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [1804ms] refreshing cache from master, [1804ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [1806ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@85502774(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@7f68074a -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@385979398 after retry RPC@1461184586) -> (continuation of Deferred@106893805 after retry RPC@1851976916) -> (continuation of Deferred@1536646205 after retry RPC@577376215) -> (continuation of Deferred@767133785 after retry RPC@1240430424) -> (continuation of Deferred@1866692936 after retry RPC@1562674767), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@74968804 -> wakeup thread Time-limited test -> (continuation of Deferred@385979398 after retry RPC@1461184586) -> (continuation of Deferred@106893805 after retry RPC@1851976916) -> (continuation of Deferred@1536646205 after retry RPC@577376215) -> (continuation of Deferred@767133785 after retry RPC@1240430424) -> (continuation of Deferred@1866692936 after retry RPC@1562674767))) after lookup
02:03:43.855 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] connecting to peer
02:03:43.857 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] Successfully connected to peer
02:03:43.877 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0xa740c605, L:/127.0.0.1:51528 - R:/127.29.165.130:35181] using TOKEN/null
02:03:43.895 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.29.165.129:45737
02:03:43.896 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

02:03:43.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:43.897439 30358 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskxPorqF/build/debug/bin/kudu
02:03:43.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskxPorqF/build/debug/bin/kudu
02:03:43.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/wal
02:03:43.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/data
02:03:43.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:03:43.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:03:43.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:03:43.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:03:43.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:03:43.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:03:43.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/logs
02:03:43.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/data/info.pb
02:03:43.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:03:43.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:03:43.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:03:43.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:03:43.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:03:43.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:03:43.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:03:43.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:03:43.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:03:43.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:03:43.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.165.129:45737
02:03:43.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.165.129
02:03:43.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.165.190:39217,127.29.165.189:39733,127.29.165.188:38583
02:03:43.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=46663
02:03:43.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.165.129
02:03:43.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.165.148:38003
02:03:43.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:03:43.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:03:43.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:03:44.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:44.002959 31312 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:03:44.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:44.003163 31312 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:03:44.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:44.003206 31312 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:03:44.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:44.006542 31312 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:03:44.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:44.006644 31312 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.165.129
02:03:44.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.010365 31312 tablet_server_runner.cc:78] Tablet server non-default flags:
02:03:44.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:03:44.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.165.148:38003
02:03:44.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:03:44.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:03:44.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/data
02:03:44.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/wal
02:03:44.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:03:44.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:03:44.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.165.129:45737
02:03:44.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:03:44.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:03:44.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:03:44.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/data/info.pb
02:03:44.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.165.129
02:03:44.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=46663
02:03:44.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.165.190:39217,127.29.165.189:39733,127.29.165.188:38583
02:03:44.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:03:44.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.31312
02:03:44.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:03:44.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:03:44.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:03:44.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:03:44.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.165.129
02:03:44.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/logs
02:03:44.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:03:44.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:03:44.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:03:44.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:03:44.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:03:44.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
02:03:44.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type DEBUG
02:03:44.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 01:43:15 UTC on bdcb31816ec0
02:03:44.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11948
02:03:44.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.011451 31312 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:03:44.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.012590 31312 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:03:44.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:44.018652 31317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:44.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:44.018731 31318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:44.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:44.018661 31320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:44.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.019608 31312 server_base.cc:1061] running on GCE node
02:03:44.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.019943 31312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:03:44.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.020476 31312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:03:44.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.021636 31312 hybrid_clock.cc:648] HybridClock initialized: now 1779069824021610 us; error 40 us; skew 500 ppm
02:03:44.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.023660 31312 webserver.cc:533] Webserver started at http://127.29.165.129:46663/ using document root <none> and password file <none>
02:03:44.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.024262 31312 fs_manager.cc:362] Metadata directory not provided
02:03:44.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.024340 31312 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:03:44.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.027904 31312 fs_manager.cc:714] Time spent opening directory manager: real 0.002s	user 0.002s	sys 0.002s
02:03:44.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.029619 31326 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:03:44.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.030597 31312 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
02:03:44.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.030680 31312 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/data,/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/wal
02:03:44.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "06f0647651a749f8a533afd451fe5fcb"
02:03:44.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:03:36 on dist-test-slave-838s"
02:03:44.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.031101 31312 fs_report.cc:389] FS layout report
02:03:44.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:03:44.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/wal
02:03:44.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/wal
02:03:44.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/data/data
02:03:44.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:03:44.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:03:44.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:03:44.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:03:44.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:03:44.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:03:44.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:03:44.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:03:44.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:03:44.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:03:44.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:03:44.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.050240 31312 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:03:44.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.050868 31312 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:03:44.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.051016 31312 kserver.cc:163] Server-wide thread pool size limit: 3276
02:03:44.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.051481 31312 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:03:44.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.052663 31333 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
02:03:44.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.055886 31312 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:03:44.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.055955 31312 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.003s	user 0.000s	sys 0.000s
02:03:44.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.056038 31312 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:03:44.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.057422 31312 ts_tablet_manager.cc:616] Registered 1 tablets
02:03:44.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.057488 31312 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s	user 0.000s	sys 0.000s
02:03:44.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.057777 31333 tablet_bootstrap.cc:492] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb: Bootstrap starting.
02:03:44.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.068787 31312 rpc_server.cc:307] RPC server started. Bound to: 127.29.165.129:45737
02:03:44.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.068814 31440 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.165.129:45737 every 8 connection(s)
02:03:44.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.069844 31312 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/data/info.pb
02:03:44.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.073139 31333 log.cc:826] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb: Log is configured to *not* fsync() on all Append() calls
02:03:44.073 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:03:44.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.073401 30358 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskxPorqF/build/debug/bin/kudu as pid 31312
02:03:44.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.081820 31441 heartbeater.cc:344] Connected to a master server at 127.29.165.190:39217
02:03:44.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.082058 31441 heartbeater.cc:461] Registering TS with master...
02:03:44.082 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 9 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956, tablet=830b8d1af48f4d34b281a548f1dbe5ef, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:03:44.083 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.29.165.130:35181
02:03:44.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.082952 31442 heartbeater.cc:344] Connected to a master server at 127.29.165.188:38583
02:03:44.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.083046 31442 heartbeater.cc:461] Registering TS with master...
02:03:44.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.083119 31441 heartbeater.cc:507] Master 127.29.165.190:39217 requested a full tablet report, sending...
02:03:44.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.083244 31442 heartbeater.cc:507] Master 127.29.165.188:38583 requested a full tablet report, sending...
02:03:44.083 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

02:03:44.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.084035 30358 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskxPorqF/build/debug/bin/kudu with pid 30775
02:03:44.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.086632 30543 ts_manager.cc:194] Re-registered known tserver with Master: 06f0647651a749f8a533afd451fe5fcb (127.29.165.129:45737)
02:03:44.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.086654 30399 ts_manager.cc:194] Re-registered known tserver with Master: 06f0647651a749f8a533afd451fe5fcb (127.29.165.129:45737)
02:03:44.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.087579 30543 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.165.129:46465
02:03:44.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.094172 31443 heartbeater.cc:344] Connected to a master server at 127.29.165.189:39733
02:03:44.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.094264 31443 heartbeater.cc:461] Registering TS with master...
02:03:44.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.094419 31443 heartbeater.cc:507] Master 127.29.165.189:39733 requested a full tablet report, sending...
02:03:44.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.095259 30471 ts_manager.cc:194] Re-registered known tserver with Master: 06f0647651a749f8a533afd451fe5fcb (127.29.165.129:45737)
02:03:44.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.095348 31333 tablet_bootstrap.cc:492] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=6 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:03:44.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.095728 31333 tablet_bootstrap.cc:492] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb: Bootstrap complete.
02:03:44.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.097086 31333 ts_tablet_manager.cc:1403] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb: Time spent bootstrapping tablet: real 0.040s	user 0.017s	sys 0.008s
02:03:44.098 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:03:44.099 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] handling channelInactive
02:03:44.100 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] cleaning up while in state READY due to: connection closed
02:03:44.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.100342 31333 raft_consensus.cc:359] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:03:44.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.100651 31333 raft_consensus.cc:740] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06f0647651a749f8a533afd451fe5fcb, State: Initialized, Role: FOLLOWER
02:03:44.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.101001 31333 consensus_queue.cc:260] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:03:44.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.101541 31442 heartbeater.cc:499] Master 127.29.165.188:38583 was elected leader, sending a full tablet report...
02:03:44.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:44.101765 31333 ts_tablet_manager.cc:1434] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb: Time spent starting tablet: real 0.005s	user 0.006s	sys 0.000s
02:03:44.111 [DEBUG - pool-2-thread-1] (Connection.java:748) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] connecting to peer
02:03:44.112 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:03:44.113 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] cleaning up while in state CONNECTING due to: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:03:44.113 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:03:44.133 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:03:44.134 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737),26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535),4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)[L]]}]
02:03:44.135 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=3, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=36), Traces: [0ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [1ms] delaying RPC due to: Illegal state: connection is terminated, [12ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [14ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [14ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [34ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@1221678028(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@47c07e79 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4ea6400d -> wakeup thread Time-limited test)) after lookup
02:03:44.136 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] connecting to peer
02:03:44.137 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:03:44.137 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] cleaning up while in state CONNECTING due to: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:03:44.137 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:03:44.155 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:03:44.156 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737),26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535),4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)[L]]}]
02:03:44.156 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=5, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=57), Traces: [0ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [1ms] delaying RPC due to: Illegal state: connection is terminated, [12ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [14ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [14ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [34ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [36ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [38ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [38ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [53ms] refreshing cache from master, [53ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [56ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@1221678028(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@47c07e79 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1363822504 after retry RPC@756073380), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4ea6400d -> wakeup thread Time-limited test -> (continuation of Deferred@1363822504 after retry RPC@756073380))) after lookup
02:03:44.157 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] connecting to peer
02:03:44.158 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:03:44.158 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] cleaning up while in state CONNECTING due to: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:03:44.158 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:03:44.195 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:03:44.195 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737),26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535),4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)[L]]}]
02:03:44.196 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=7, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=97), Traces: [0ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [1ms] delaying RPC due to: Illegal state: connection is terminated, [12ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [14ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [14ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [34ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [36ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [38ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [38ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [53ms] refreshing cache from master, [53ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [56ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [57ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [59ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [59ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [92ms] refreshing cache from master, [93ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [96ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@1221678028(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@47c07e79 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1363822504 after retry RPC@756073380) -> (continuation of Deferred@124791542 after retry RPC@743883871), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4ea6400d -> wakeup thread Time-limited test -> (continuation of Deferred@1363822504 after retry RPC@756073380) -> (continuation of Deferred@124791542 after retry RPC@743883871))) after lookup
02:03:44.197 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] connecting to peer
02:03:44.199 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:03:44.199 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] cleaning up while in state CONNECTING due to: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:03:44.199 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:03:44.334 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:03:44.335 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737),26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535),4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)[L]]}]
02:03:44.335 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=9, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=236), Traces: [0ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [1ms] delaying RPC due to: Illegal state: connection is terminated, [12ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [14ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [14ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [34ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [36ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [38ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [38ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [53ms] refreshing cache from master, [53ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [56ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [57ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [59ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [59ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [92ms] refreshing cache from master, [93ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [96ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [98ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [100ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [100ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [232ms] refreshing cache from master, [232ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [235ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@1221678028(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@47c07e79 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1363822504 after retry RPC@756073380) -> (continuation of Deferred@124791542 after retry RPC@743883871) -> (continuation of Deferred@1458355131 after retry RPC@1275263237), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4ea6400d -> wakeup thread Time-limited test -> (continuation of Deferred@1363822504 after retry RPC@756073380) -> (continuation of Deferred@124791542 after retry RPC@743883871) -> (continuation of Deferred@1458355131 after retry RPC@1275263237))) after lookup
02:03:44.336 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] connecting to peer
02:03:44.337 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:03:44.337 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] cleaning up while in state CONNECTING due to: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:03:44.337 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:03:44.374 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:03:44.374 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737),26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535),4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)[L]]}]
02:03:44.375 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=11, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=276), Traces: [0ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [1ms] delaying RPC due to: Illegal state: connection is terminated, [12ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [14ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [14ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [34ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [36ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [38ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [38ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [53ms] refreshing cache from master, [53ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [56ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [57ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [59ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [59ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [92ms] refreshing cache from master, [93ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [96ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [98ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [100ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [100ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [232ms] refreshing cache from master, [232ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [235ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [236ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [238ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [238ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [272ms] refreshing cache from master, [272ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [275ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@1221678028(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@47c07e79 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1363822504 after retry RPC@756073380) -> (continuation of Deferred@124791542 after retry RPC@743883871) -> (continuation of Deferred@1458355131 after retry RPC@1275263237) -> (continuation of Deferred@2072255357 after retry RPC@546639833), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4ea6400d -> wakeup thread Time-limited test -> (continuation of Deferred@1363822504 after retry RPC@756073380) -> (continuation of Deferred@124791542 after retry RPC@743883871) -> (continuation of Deferred@1458355131 after retry RPC@1275263237) -> (continuation of Deferred@2072255357 after retry RPC@546639833))) after lookup
02:03:44.375 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] connecting to peer
02:03:44.376 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:03:44.376 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] cleaning up while in state CONNECTING due to: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:03:44.377 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:03:45.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:45.394886 31455 raft_consensus.cc:493] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4e39ec54c2d642feafa81dc5d4f066c2)
02:03:45.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:45.395028 31455 raft_consensus.cc:515] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:03:45.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:45.395465 31455 leader_election.cc:290] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130:35181), 06f0647651a749f8a533afd451fe5fcb (127.29.165.129:45737)
02:03:45.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:45.396816 31155 leader_election.cc:336] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130:35181): Network error: Client connection negotiation failed: client connection to 127.29.165.130:35181: connect: Connection refused (error 111)
02:03:45.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:45.400439 31395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "830b8d1af48f4d34b281a548f1dbe5ef" candidate_uuid: "26ea15144933497a98a05f83b05e8f93" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "06f0647651a749f8a533afd451fe5fcb" is_pre_election: true
02:03:45.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:45.400763 31395 raft_consensus.cc:2468] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26ea15144933497a98a05f83b05e8f93 in term 2.
02:03:45.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:45.401539 31156 leader_election.cc:304] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06f0647651a749f8a533afd451fe5fcb, 26ea15144933497a98a05f83b05e8f93; no voters: 4e39ec54c2d642feafa81dc5d4f066c2
02:03:45.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:45.401723 31455 raft_consensus.cc:2804] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 3 FOLLOWER]: Leader pre-election won for term 4
02:03:45.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:45.401779 31455 raft_consensus.cc:493] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4e39ec54c2d642feafa81dc5d4f066c2)
02:03:45.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:45.401825 31455 raft_consensus.cc:3060] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 3 FOLLOWER]: Advancing to term 4
02:03:45.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:45.402607 31455 raft_consensus.cc:515] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:03:45.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:45.403064 31455 leader_election.cc:290] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [CANDIDATE]: Term 4 election: Requested vote from peers 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130:35181), 06f0647651a749f8a533afd451fe5fcb (127.29.165.129:45737)
02:03:45.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:45.403502 31395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "830b8d1af48f4d34b281a548f1dbe5ef" candidate_uuid: "26ea15144933497a98a05f83b05e8f93" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "06f0647651a749f8a533afd451fe5fcb"
02:03:45.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:45.403638 31395 raft_consensus.cc:3060] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 2 FOLLOWER]: Advancing to term 4
02:03:45.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:45.404156 31155 leader_election.cc:336] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130:35181): Network error: Client connection negotiation failed: client connection to 127.29.165.130:35181: connect: Connection refused (error 111)
02:03:45.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:45.405371 31395 raft_consensus.cc:2468] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26ea15144933497a98a05f83b05e8f93 in term 4.
02:03:45.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:45.405678 31156 leader_election.cc:304] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06f0647651a749f8a533afd451fe5fcb, 26ea15144933497a98a05f83b05e8f93; no voters: 4e39ec54c2d642feafa81dc5d4f066c2
02:03:45.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:45.405828 31455 raft_consensus.cc:2804] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 4 FOLLOWER]: Leader election won for term 4
02:03:45.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:45.406032 31455 raft_consensus.cc:697] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 4 LEADER]: Becoming Leader. State: Replica: 26ea15144933497a98a05f83b05e8f93, State: Running, Role: LEADER
02:03:45.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:45.406340 31455 consensus_queue.cc:237] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:03:45.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:45.408814 30543 catalog_manager.cc:5696] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 reported cstate change: term changed from 3 to 4, leader changed from 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130) to 26ea15144933497a98a05f83b05e8f93 (127.29.165.131). New cstate: current_term: 4 leader_uuid: "26ea15144933497a98a05f83b05e8f93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } health_report { overall_health: HEALTHY } } }
02:03:45.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:45.884322 31395 raft_consensus.cc:1275] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 4 FOLLOWER]: Refusing update from remote peer 26ea15144933497a98a05f83b05e8f93: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 4 index: 13. (index mismatch)
02:03:45.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:45.884969 31455 consensus_queue.cc:1048] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 8, Time since last communication: 0.000s
02:03:45.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:45.893375 31155 consensus_peers.cc:597] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 -> Peer 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130:35181): Couldn't send request to peer 4e39ec54c2d642feafa81dc5d4f066c2. Status: Network error: Client connection negotiation failed: client connection to 127.29.165.130:35181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:03:47.794 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:03:47.795 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737),26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)[L],4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)]}]
02:03:47.795 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=13, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=3696), Traces: [0ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [1ms] delaying RPC due to: Illegal state: connection is terminated, [12ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [14ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [14ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [34ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [36ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [38ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [38ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [53ms] refreshing cache from master, [53ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [56ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [57ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [59ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [59ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [92ms] refreshing cache from master, [93ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [96ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [98ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [100ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [100ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [232ms] refreshing cache from master, [232ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [235ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [236ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [238ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [238ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [272ms] refreshing cache from master, [272ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [275ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [276ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [278ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [278ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [3692ms] refreshing cache from master, [3692ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [3695ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@1221678028(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@47c07e79 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1363822504 after retry RPC@756073380) -> (continuation of Deferred@124791542 after retry RPC@743883871) -> (continuation of Deferred@1458355131 after retry RPC@1275263237) -> (continuation of Deferred@2072255357 after retry RPC@546639833) -> (continuation of Deferred@1500068707 after retry RPC@1368001616), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4ea6400d -> wakeup thread Time-limited test -> (continuation of Deferred@1363822504 after retry RPC@756073380) -> (continuation of Deferred@124791542 after retry RPC@743883871) -> (continuation of Deferred@1458355131 after retry RPC@1275263237) -> (continuation of Deferred@2072255357 after retry RPC@546639833) -> (continuation of Deferred@1500068707 after retry RPC@1368001616))) after lookup
02:03:47.796 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] connecting to peer
02:03:47.797 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] Successfully connected to peer
02:03:47.811 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0xbed95b80, L:/127.0.0.1:47058 - R:/127.29.165.131:44535] using TOKEN/null
02:03:47.829 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.29.165.130:35181
02:03:47.829 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

02:03:47.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:47.830112 30358 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskxPorqF/build/debug/bin/kudu
02:03:47.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskxPorqF/build/debug/bin/kudu
02:03:47.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/wal
02:03:47.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/data
02:03:47.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:03:47.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:03:47.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:03:47.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:03:47.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:03:47.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:03:47.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/logs
02:03:47.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/data/info.pb
02:03:47.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:03:47.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:03:47.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:03:47.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:03:47.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:03:47.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:03:47.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:03:47.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:03:47.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:03:47.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:03:47.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.165.130:35181
02:03:47.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.165.130
02:03:47.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.165.190:39217,127.29.165.189:39733,127.29.165.188:38583
02:03:47.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=33955
02:03:47.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.165.130
02:03:47.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.165.148:38003
02:03:47.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:03:47.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:03:47.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:03:47.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:47.936076 31496 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:03:47.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:47.936295 31496 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:03:47.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:47.936352 31496 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:03:47.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:47.939793 31496 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:03:47.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:47.939934 31496 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.165.130
02:03:47.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:47.943660 31496 tablet_server_runner.cc:78] Tablet server non-default flags:
02:03:47.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:03:47.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.165.148:38003
02:03:47.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:03:47.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:03:47.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/data
02:03:47.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/wal
02:03:47.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:03:47.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:03:47.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.165.130:35181
02:03:47.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:03:47.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:03:47.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:03:47.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/data/info.pb
02:03:47.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.165.130
02:03:47.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=33955
02:03:47.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.165.190:39217,127.29.165.189:39733,127.29.165.188:38583
02:03:47.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:03:47.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.31496
02:03:47.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:03:47.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:03:47.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:03:47.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:03:47.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.165.130
02:03:47.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/logs
02:03:47.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:03:47.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:03:47.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:03:47.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:03:47.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:03:47.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
02:03:47.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type DEBUG
02:03:47.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 01:43:15 UTC on bdcb31816ec0
02:03:47.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11948
02:03:47.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:47.944981 31496 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:03:47.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:47.946128 31496 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:03:47.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:47.952188 31501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:47.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:47.952467 31504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:47.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:47.952639 31502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:47.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:47.953040 31496 server_base.cc:1061] running on GCE node
02:03:47.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:47.953450 31496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:03:47.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:47.954124 31496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:03:47.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:47.955309 31496 hybrid_clock.cc:648] HybridClock initialized: now 1779069827955293 us; error 51 us; skew 500 ppm
02:03:47.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:47.957779 31496 webserver.cc:533] Webserver started at http://127.29.165.130:33955/ using document root <none> and password file <none>
02:03:47.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:47.958508 31496 fs_manager.cc:362] Metadata directory not provided
02:03:47.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:47.958599 31496 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:03:47.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:47.963727 31496 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.002s	sys 0.001s
02:03:47.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:47.965777 31510 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:03:47.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:47.966823 31496 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
02:03:47.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:47.966936 31496 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/data,/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/wal
02:03:47.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "4e39ec54c2d642feafa81dc5d4f066c2"
02:03:47.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:03:36 on dist-test-slave-838s"
02:03:47.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:47.967357 31496 fs_report.cc:389] FS layout report
02:03:47.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:03:47.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/wal
02:03:47.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/wal
02:03:47.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/data/data
02:03:47.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:03:47.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:03:47.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:03:47.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:03:47.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:03:47.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:03:47.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:03:47.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:03:47.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:03:47.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:03:47.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:03:48.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:48.000268 31496 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:03:48.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:48.001015 31496 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:03:48.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:48.001219 31496 kserver.cc:163] Server-wide thread pool size limit: 3276
02:03:48.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:48.001788 31496 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:03:48.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:48.003077 31517 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
02:03:48.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:48.006397 31496 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:03:48.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:48.006469 31496 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.004s	user 0.000s	sys 0.000s
02:03:48.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:48.006531 31496 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:03:48.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:48.007972 31496 ts_tablet_manager.cc:616] Registered 1 tablets
02:03:48.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:48.008045 31496 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s	user 0.000s	sys 0.000s
02:03:48.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:48.008327 31517 tablet_bootstrap.cc:492] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2: Bootstrap starting.
02:03:48.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:48.018777 31496 rpc_server.cc:307] RPC server started. Bound to: 127.29.165.130:35181
02:03:48.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:48.018827 31624 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.165.130:35181 every 8 connection(s)
02:03:48.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:48.019863 31496 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/data/info.pb
02:03:48.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:48.024601 31517 log.cc:826] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2: Log is configured to *not* fsync() on all Append() calls
02:03:48.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:48.025741 30358 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskxPorqF/build/debug/bin/kudu as pid 31496
02:03:48.026 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:03:48.034 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 12 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956, tablet=830b8d1af48f4d34b281a548f1dbe5ef, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:03:48.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:48.035889 31626 heartbeater.cc:344] Connected to a master server at 127.29.165.188:38583
02:03:48.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:48.036139 31626 heartbeater.cc:461] Registering TS with master...
02:03:48.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:48.036710 31626 heartbeater.cc:507] Master 127.29.165.188:38583 requested a full tablet report, sending...
02:03:48.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:48.038633 30543 ts_manager.cc:194] Re-registered known tserver with Master: 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130:35181)
02:03:48.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:48.039777 30543 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.165.130:40035
02:03:48.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:48.044899 31625 heartbeater.cc:344] Connected to a master server at 127.29.165.190:39217
02:03:48.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:48.044999 31625 heartbeater.cc:461] Registering TS with master...
02:03:48.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:48.045171 31625 heartbeater.cc:507] Master 127.29.165.190:39217 requested a full tablet report, sending...
02:03:48.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:48.046509 30400 ts_manager.cc:194] Re-registered known tserver with Master: 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130:35181)
02:03:48.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:48.046579 31517 tablet_bootstrap.cc:492] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=9 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:03:48.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:48.046993 31517 tablet_bootstrap.cc:492] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2: Bootstrap complete.
02:03:48.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:48.048316 31517 ts_tablet_manager.cc:1403] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2: Time spent bootstrapping tablet: real 0.040s	user 0.017s	sys 0.012s
02:03:48.048 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.29.165.131:44535
02:03:48.048 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

02:03:48.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:48.049060 30358 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskxPorqF/build/debug/bin/kudu with pid 31138
02:03:48.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:48.049132 31627 heartbeater.cc:344] Connected to a master server at 127.29.165.189:39733
02:03:48.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:48.049202 31627 heartbeater.cc:461] Registering TS with master...
02:03:48.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:48.049311 31627 heartbeater.cc:507] Master 127.29.165.189:39733 requested a full tablet report, sending...
02:03:48.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:48.050509 30472 ts_manager.cc:194] Re-registered known tserver with Master: 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130:35181)
02:03:48.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:48.051659 31517 raft_consensus.cc:359] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:03:48.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:48.052014 31517 raft_consensus.cc:740] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e39ec54c2d642feafa81dc5d4f066c2, State: Initialized, Role: FOLLOWER
02:03:48.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:48.052462 31517 consensus_queue.cc:260] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:03:48.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:48.053242 31517 ts_tablet_manager.cc:1434] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.001s
02:03:48.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:48.053922 31626 heartbeater.cc:499] Master 127.29.165.188:38583 was elected leader, sending a full tablet report...
02:03:48.059 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] handling channelInactive
02:03:48.059 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] cleaning up while in state READY due to: connection closed
02:03:48.059 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:03:48.060 [DEBUG - Time-limited test] (Connection.java:748) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] connecting to peer
02:03:48.061 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:48.061 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] cleaning up while in state CONNECTING due to: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:48.061 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:48.074 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:03:48.074 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737),26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)[L],4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)]}]
02:03:48.075 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=2, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=15), Traces: [0ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [2ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [2ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [15ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@188705279(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@69ba17f9 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@f73501e -> wakeup thread Time-limited test)) after lookup
02:03:48.076 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] connecting to peer
02:03:48.077 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:48.077 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] cleaning up while in state CONNECTING due to: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:48.077 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:48.093 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:03:48.094 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737),26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)[L],4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)]}]
02:03:48.094 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=4, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=34), Traces: [0ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [2ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [2ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [15ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [16ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [18ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [18ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [34ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@188705279(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@69ba17f9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@661895748 after retry RPC@147487680), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@f73501e -> wakeup thread Time-limited test -> (continuation of Deferred@661895748 after retry RPC@147487680))) after lookup
02:03:48.095 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] connecting to peer
02:03:48.096 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:48.096 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] cleaning up while in state CONNECTING due to: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:48.096 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:48.135 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:03:48.135 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737),26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)[L],4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)]}]
02:03:48.136 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=6, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=76), Traces: [0ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [2ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [2ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [15ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [16ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [18ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [18ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [34ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [35ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [37ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [37ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [72ms] refreshing cache from master, [72ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [75ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@188705279(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@69ba17f9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@661895748 after retry RPC@147487680) -> (continuation of Deferred@359109700 after retry RPC@1311304019), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@f73501e -> wakeup thread Time-limited test -> (continuation of Deferred@661895748 after retry RPC@147487680) -> (continuation of Deferred@359109700 after retry RPC@1311304019))) after lookup
02:03:48.137 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] connecting to peer
02:03:48.138 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:48.138 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] cleaning up while in state CONNECTING due to: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:48.138 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:48.154 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:03:48.154 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737),26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)[L],4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)]}]
02:03:48.154 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=8, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=95), Traces: [0ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [2ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [2ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [15ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [16ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [18ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [18ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [34ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [35ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [37ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [37ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [72ms] refreshing cache from master, [72ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [75ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [77ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [79ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [79ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [92ms] refreshing cache from master, [92ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [94ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@188705279(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@69ba17f9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@661895748 after retry RPC@147487680) -> (continuation of Deferred@359109700 after retry RPC@1311304019) -> (continuation of Deferred@1141925008 after retry RPC@1308327728), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@f73501e -> wakeup thread Time-limited test -> (continuation of Deferred@661895748 after retry RPC@147487680) -> (continuation of Deferred@359109700 after retry RPC@1311304019) -> (continuation of Deferred@1141925008 after retry RPC@1308327728))) after lookup
02:03:48.155 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] connecting to peer
02:03:48.156 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:48.157 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] cleaning up while in state CONNECTING due to: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:48.157 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:48.454 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:03:48.454 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737),26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)[L],4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)]}]
02:03:48.455 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=10, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=395), Traces: [0ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [2ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [2ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [15ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [16ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [18ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [18ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [34ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [35ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [37ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [37ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [72ms] refreshing cache from master, [72ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [75ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [77ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [79ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [79ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [92ms] refreshing cache from master, [92ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [94ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [96ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [98ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [98ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [392ms] refreshing cache from master, [392ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [395ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@188705279(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@69ba17f9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@661895748 after retry RPC@147487680) -> (continuation of Deferred@359109700 after retry RPC@1311304019) -> (continuation of Deferred@1141925008 after retry RPC@1308327728) -> (continuation of Deferred@1765939152 after retry RPC@2136405815), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@f73501e -> wakeup thread Time-limited test -> (continuation of Deferred@661895748 after retry RPC@147487680) -> (continuation of Deferred@359109700 after retry RPC@1311304019) -> (continuation of Deferred@1141925008 after retry RPC@1308327728) -> (continuation of Deferred@1765939152 after retry RPC@2136405815))) after lookup
02:03:48.456 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] connecting to peer
02:03:48.457 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:48.457 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] cleaning up while in state CONNECTING due to: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:48.457 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:49.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.295358 31638 raft_consensus.cc:493] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
02:03:49.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.295534 31638 raft_consensus.cc:515] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:03:49.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.296618 31638 leader_election.cc:290] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 06f0647651a749f8a533afd451fe5fcb (127.29.165.129:45737), 26ea15144933497a98a05f83b05e8f93 (127.29.165.131:44535)
02:03:49.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:49.297336 31513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.165.131:44535: connect: Connection refused (error 111)
02:03:49.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:49.298170 31513 leader_election.cc:336] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 26ea15144933497a98a05f83b05e8f93 (127.29.165.131:44535): Network error: Client connection negotiation failed: client connection to 127.29.165.131:44535: connect: Connection refused (error 111)
02:03:49.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.300984 31395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "830b8d1af48f4d34b281a548f1dbe5ef" candidate_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "06f0647651a749f8a533afd451fe5fcb" is_pre_election: true
02:03:49.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.301503 31514 leader_election.cc:304] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e39ec54c2d642feafa81dc5d4f066c2; no voters: 06f0647651a749f8a533afd451fe5fcb, 26ea15144933497a98a05f83b05e8f93
02:03:49.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.301731 31638 raft_consensus.cc:2749] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
02:03:49.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.329236 31643 raft_consensus.cc:493] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 26ea15144933497a98a05f83b05e8f93)
02:03:49.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.329389 31643 raft_consensus.cc:515] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:03:49.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.330386 31643 leader_election.cc:290] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130:35181), 26ea15144933497a98a05f83b05e8f93 (127.29.165.131:44535)
02:03:49.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:49.331540 31329 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.165.131:44535: connect: Connection refused (error 111)
02:03:49.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:49.332489 31329 leader_election.cc:336] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 26ea15144933497a98a05f83b05e8f93 (127.29.165.131:44535): Network error: Client connection negotiation failed: client connection to 127.29.165.131:44535: connect: Connection refused (error 111)
02:03:49.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.335352 31579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "830b8d1af48f4d34b281a548f1dbe5ef" candidate_uuid: "06f0647651a749f8a533afd451fe5fcb" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" is_pre_election: true
02:03:49.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.335654 31579 raft_consensus.cc:2468] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06f0647651a749f8a533afd451fe5fcb in term 3.
02:03:49.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.336585 31329 leader_election.cc:304] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06f0647651a749f8a533afd451fe5fcb, 4e39ec54c2d642feafa81dc5d4f066c2; no voters: 26ea15144933497a98a05f83b05e8f93
02:03:49.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.336872 31643 raft_consensus.cc:2804] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 4 FOLLOWER]: Leader pre-election won for term 5
02:03:49.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.336941 31643 raft_consensus.cc:493] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 4 FOLLOWER]: Starting leader election (detected failure of leader 26ea15144933497a98a05f83b05e8f93)
02:03:49.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.337023 31643 raft_consensus.cc:3060] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 4 FOLLOWER]: Advancing to term 5
02:03:49.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.338210 31643 raft_consensus.cc:515] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:03:49.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.338646 31643 leader_election.cc:290] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [CANDIDATE]: Term 5 election: Requested vote from peers 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130:35181), 26ea15144933497a98a05f83b05e8f93 (127.29.165.131:44535)
02:03:49.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.338972 31579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "830b8d1af48f4d34b281a548f1dbe5ef" candidate_uuid: "06f0647651a749f8a533afd451fe5fcb" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "4e39ec54c2d642feafa81dc5d4f066c2"
02:03:49.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.339095 31579 raft_consensus.cc:3060] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 3 FOLLOWER]: Advancing to term 5
02:03:49.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:49.340503 31329 leader_election.cc:336] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 26ea15144933497a98a05f83b05e8f93 (127.29.165.131:44535): Network error: Client connection negotiation failed: client connection to 127.29.165.131:44535: connect: Connection refused (error 111)
02:03:49.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.341281 31579 raft_consensus.cc:2468] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06f0647651a749f8a533afd451fe5fcb in term 5.
02:03:49.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.341655 31329 leader_election.cc:304] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06f0647651a749f8a533afd451fe5fcb, 4e39ec54c2d642feafa81dc5d4f066c2; no voters: 26ea15144933497a98a05f83b05e8f93
02:03:49.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.341876 31643 raft_consensus.cc:2804] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 5 FOLLOWER]: Leader election won for term 5
02:03:49.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.342089 31643 raft_consensus.cc:697] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 5 LEADER]: Becoming Leader. State: Replica: 06f0647651a749f8a533afd451fe5fcb, State: Running, Role: LEADER
02:03:49.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.342345 31643 consensus_queue.cc:237] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 4.16, Last appended by leader: 16, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:03:49.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.344874 30543 catalog_manager.cc:5696] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb reported cstate change: term changed from 4 to 5, leader changed from 26ea15144933497a98a05f83b05e8f93 (127.29.165.131) to 06f0647651a749f8a533afd451fe5fcb (127.29.165.129). New cstate: current_term: 5 leader_uuid: "06f0647651a749f8a533afd451fe5fcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } health_report { overall_health: UNKNOWN } } }
02:03:49.674 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:03:49.675 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)[L],26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535),4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)]}]
02:03:49.675 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=12, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=1615), Traces: [0ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [2ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [2ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [15ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [16ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [18ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [18ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [34ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [35ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [37ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [37ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [72ms] refreshing cache from master, [72ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [75ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [77ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [79ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [79ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [92ms] refreshing cache from master, [92ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [94ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [96ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [98ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [98ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [392ms] refreshing cache from master, [392ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [395ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [396ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [398ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [398ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [1612ms] refreshing cache from master, [1612ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [1615ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@188705279(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@69ba17f9 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@661895748 after retry RPC@147487680) -> (continuation of Deferred@359109700 after retry RPC@1311304019) -> (continuation of Deferred@1141925008 after retry RPC@1308327728) -> (continuation of Deferred@1765939152 after retry RPC@2136405815) -> (continuation of Deferred@832954467 after retry RPC@2030196271), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@f73501e -> wakeup thread Time-limited test -> (continuation of Deferred@661895748 after retry RPC@147487680) -> (continuation of Deferred@359109700 after retry RPC@1311304019) -> (continuation of Deferred@1141925008 after retry RPC@1308327728) -> (continuation of Deferred@1765939152 after retry RPC@2136405815) -> (continuation of Deferred@832954467 after retry RPC@2030196271))) after lookup
02:03:49.676 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] connecting to peer
02:03:49.677 [DEBUG - kudu-nio-7] (Connection.java:757) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] Successfully connected to peer
02:03:49.696 [DEBUG - kudu-nio-7] (Negotiator.java:919) Authenticated connection [id: 0x5234c1ac, L:/127.0.0.1:40512 - R:/127.29.165.129:45737] using TOKEN/null
02:03:49.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:49.700095 31329 consensus_peers.cc:597] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb -> Peer 26ea15144933497a98a05f83b05e8f93 (127.29.165.131:44535): Couldn't send request to peer 26ea15144933497a98a05f83b05e8f93. Status: Network error: Client connection negotiation failed: client connection to 127.29.165.131:44535: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:03:49.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.700160 31579 raft_consensus.cc:1275] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 5 FOLLOWER]: Refusing update from remote peer 06f0647651a749f8a533afd451fe5fcb: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 5 index: 18. (index mismatch)
02:03:49.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.700726 31643 consensus_queue.cc:1048] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 12, Time since last communication: 0.000s
02:03:49.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.706388 31661 mvcc.cc:204] Tried to move back new op lower bound from 7287070022444863488 to 7287070020989882368. Current Snapshot: MvccSnapshot[applied={T|T < 7287070022444863488}]
02:03:49.720 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.29.165.131:44535
02:03:49.720 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

02:03:49.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.720885 30358 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskxPorqF/build/debug/bin/kudu
02:03:49.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskxPorqF/build/debug/bin/kudu
02:03:49.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/wal
02:03:49.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/data
02:03:49.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:03:49.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:03:49.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:03:49.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:03:49.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:03:49.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:03:49.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/logs
02:03:49.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/data/info.pb
02:03:49.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:03:49.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:03:49.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:03:49.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:03:49.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:03:49.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:03:49.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:03:49.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:03:49.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:03:49.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:03:49.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.165.131:44535
02:03:49.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.165.131
02:03:49.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.165.190:39217,127.29.165.189:39733,127.29.165.188:38583
02:03:49.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45365
02:03:49.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.165.131
02:03:49.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.165.148:38003
02:03:49.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:03:49.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:03:49.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:03:49.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:49.820876 31669 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:03:49.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:49.821061 31669 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:03:49.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:49.821105 31669 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:03:49.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:49.824270 31669 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:03:49.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:49.824360 31669 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.165.131
02:03:49.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.828018 31669 tablet_server_runner.cc:78] Tablet server non-default flags:
02:03:49.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:03:49.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.165.148:38003
02:03:49.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:03:49.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:03:49.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/data
02:03:49.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/wal
02:03:49.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:03:49.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:03:49.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.165.131:44535
02:03:49.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:03:49.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:03:49.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:03:49.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/data/info.pb
02:03:49.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.165.131
02:03:49.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45365
02:03:49.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.165.190:39217,127.29.165.189:39733,127.29.165.188:38583
02:03:49.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:03:49.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.31669
02:03:49.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:03:49.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:03:49.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:03:49.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:03:49.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.165.131
02:03:49.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/logs
02:03:49.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:03:49.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:03:49.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:03:49.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:03:49.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:03:49.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
02:03:49.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type DEBUG
02:03:49.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 01:43:15 UTC on bdcb31816ec0
02:03:49.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11948
02:03:49.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.828926 31669 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:03:49.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.829967 31669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:03:49.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:49.836166 31674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:49.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.836338 31669 server_base.cc:1061] running on GCE node
02:03:49.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:49.836220 31677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:49.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:49.836215 31675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:49.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.837288 31669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:03:49.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.837734 31669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:03:49.839 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.838924 31669 hybrid_clock.cc:648] HybridClock initialized: now 1779069829838894 us; error 44 us; skew 500 ppm
02:03:49.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.840646 31669 webserver.cc:533] Webserver started at http://127.29.165.131:45365/ using document root <none> and password file <none>
02:03:49.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.841223 31669 fs_manager.cc:362] Metadata directory not provided
02:03:49.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.841308 31669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:03:49.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.844543 31669 fs_manager.cc:714] Time spent opening directory manager: real 0.002s	user 0.002s	sys 0.001s
02:03:49.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.846473 31683 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:03:49.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.847443 31669 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
02:03:49.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.847548 31669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/data,/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/wal
02:03:49.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "26ea15144933497a98a05f83b05e8f93"
02:03:49.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:03:36 on dist-test-slave-838s"
02:03:49.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.847889 31669 fs_report.cc:389] FS layout report
02:03:49.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:03:49.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/wal
02:03:49.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/wal
02:03:49.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/data/data
02:03:49.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:03:49.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:03:49.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:03:49.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:03:49.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:03:49.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:03:49.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:03:49.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:03:49.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:03:49.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:03:49.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:03:49.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.869306 31669 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:03:49.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.869889 31669 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:03:49.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.870035 31669 kserver.cc:163] Server-wide thread pool size limit: 3276
02:03:49.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.870530 31669 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:03:49.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.871798 31690 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
02:03:49.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.875140 31669 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:03:49.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.875198 31669 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.004s	user 0.001s	sys 0.000s
02:03:49.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.875244 31669 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:03:49.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.876616 31669 ts_tablet_manager.cc:616] Registered 1 tablets
02:03:49.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.876670 31669 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s	user 0.000s	sys 0.000s
02:03:49.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.876986 31690 tablet_bootstrap.cc:492] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93: Bootstrap starting.
02:03:49.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.888607 31669 rpc_server.cc:307] RPC server started. Bound to: 127.29.165.131:44535
02:03:49.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.888620 31797 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.165.131:44535 every 8 connection(s)
02:03:49.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.889986 31669 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/data/info.pb
02:03:49.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.895514 31690 log.cc:826] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93: Log is configured to *not* fsync() on all Append() calls
02:03:49.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.896354 30358 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskxPorqF/build/debug/bin/kudu as pid 31669
02:03:49.897 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:03:49.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.899858 31800 heartbeater.cc:344] Connected to a master server at 127.29.165.189:39733
02:03:49.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.900112 31800 heartbeater.cc:461] Registering TS with master...
02:03:49.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.900691 31800 heartbeater.cc:507] Master 127.29.165.189:39733 requested a full tablet report, sending...
02:03:49.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.902177 31799 heartbeater.cc:344] Connected to a master server at 127.29.165.188:38583
02:03:49.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.902271 31799 heartbeater.cc:461] Registering TS with master...
02:03:49.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.902498 31799 heartbeater.cc:507] Master 127.29.165.188:38583 requested a full tablet report, sending...
02:03:49.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.903430 30472 ts_manager.cc:194] Re-registered known tserver with Master: 26ea15144933497a98a05f83b05e8f93 (127.29.165.131:44535)
02:03:49.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.904610 30542 ts_manager.cc:194] Re-registered known tserver with Master: 26ea15144933497a98a05f83b05e8f93 (127.29.165.131:44535)
02:03:49.906 [DEBUG - kudu-nio-7] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 15 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956, tablet=830b8d1af48f4d34b281a548f1dbe5ef, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:03:49.908 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.29.165.129:45737
02:03:49.909 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

02:03:49.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.905453 30542 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.165.131:46267
02:03:49.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.906098 31798 heartbeater.cc:344] Connected to a master server at 127.29.165.190:39217
02:03:49.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.906183 31798 heartbeater.cc:461] Registering TS with master...
02:03:49.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.906363 31798 heartbeater.cc:507] Master 127.29.165.190:39217 requested a full tablet report, sending...
02:03:49.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.909433 30358 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskxPorqF/build/debug/bin/kudu with pid 31312
02:03:49.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.909725 30400 ts_manager.cc:194] Re-registered known tserver with Master: 26ea15144933497a98a05f83b05e8f93 (127.29.165.131:44535)
02:03:49.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.912222 31690 tablet_bootstrap.cc:492] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=12 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:03:49.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.912622 31690 tablet_bootstrap.cc:492] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93: Bootstrap complete.
02:03:49.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.913643 31690 ts_tablet_manager.cc:1403] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93: Time spent bootstrapping tablet: real 0.037s	user 0.032s	sys 0.000s
02:03:49.916 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:03:49.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.916064 31690 raft_consensus.cc:359] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:03:49.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.916424 31690 raft_consensus.cc:740] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26ea15144933497a98a05f83b05e8f93, State: Initialized, Role: FOLLOWER
02:03:49.916 [DEBUG - kudu-nio-7] (Connection.java:239) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] handling channelInactive
02:03:49.916 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] cleaning up while in state READY due to: connection closed
02:03:49.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.916916 31690 consensus_queue.cc:260] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 4.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:03:49.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.917575 31799 heartbeater.cc:499] Master 127.29.165.188:38583 was elected leader, sending a full tablet report...
02:03:49.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:49.917794 31690 ts_tablet_manager.cc:1434] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93: Time spent starting tablet: real 0.004s	user 0.006s	sys 0.000s
02:03:49.932 [DEBUG - pool-2-thread-1] (Connection.java:748) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] connecting to peer
02:03:49.933 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:03:49.933 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] cleaning up while in state CONNECTING due to: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:03:49.933 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:03:49.954 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:03:49.954 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)[L],26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535),4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)]}]
02:03:49.955 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=3, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=38), Traces: [0ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [0ms] delaying RPC due to: Illegal state: connection is terminated, [15ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [17ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [17ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [35ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [38ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@1357265010(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@3bbebf6f -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@276385a0 -> wakeup thread Time-limited test)) after lookup
02:03:49.955 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] connecting to peer
02:03:49.956 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:03:49.957 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] cleaning up while in state CONNECTING due to: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:03:49.957 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:03:49.974 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:03:49.974 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)[L],26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535),4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)]}]
02:03:49.974 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=5, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=58), Traces: [0ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [0ms] delaying RPC due to: Illegal state: connection is terminated, [15ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [17ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [17ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [35ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [38ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [39ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [41ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [41ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [55ms] refreshing cache from master, [55ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [57ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@1357265010(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@3bbebf6f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1520591568 after retry RPC@1270225783), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@276385a0 -> wakeup thread Time-limited test -> (continuation of Deferred@1520591568 after retry RPC@1270225783))) after lookup
02:03:49.975 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] connecting to peer
02:03:49.976 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:03:49.976 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] cleaning up while in state CONNECTING due to: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:03:49.976 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:03:50.014 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:03:50.015 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)[L],26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535),4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)]}]
02:03:50.015 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=7, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=99), Traces: [0ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [0ms] delaying RPC due to: Illegal state: connection is terminated, [15ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [17ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [17ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [35ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [38ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [39ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [41ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [41ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [55ms] refreshing cache from master, [55ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [57ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [59ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [60ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [60ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [96ms] refreshing cache from master, [96ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [98ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@1357265010(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@3bbebf6f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1520591568 after retry RPC@1270225783) -> (continuation of Deferred@398920376 after retry RPC@1817298580), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@276385a0 -> wakeup thread Time-limited test -> (continuation of Deferred@1520591568 after retry RPC@1270225783) -> (continuation of Deferred@398920376 after retry RPC@1817298580))) after lookup
02:03:50.016 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] connecting to peer
02:03:50.017 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:03:50.017 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] cleaning up while in state CONNECTING due to: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:03:50.018 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:03:50.094 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:03:50.095 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)[L],26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535),4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)]}]
02:03:50.095 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=9, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=178), Traces: [0ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [0ms] delaying RPC due to: Illegal state: connection is terminated, [15ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [17ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [17ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [35ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [38ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [39ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [41ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [41ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [55ms] refreshing cache from master, [55ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [57ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [59ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [60ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [60ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [96ms] refreshing cache from master, [96ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [98ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [100ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [101ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [102ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [175ms] refreshing cache from master, [176ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [178ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@1357265010(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@3bbebf6f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1520591568 after retry RPC@1270225783) -> (continuation of Deferred@398920376 after retry RPC@1817298580) -> (continuation of Deferred@676354679 after retry RPC@1247239311), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@276385a0 -> wakeup thread Time-limited test -> (continuation of Deferred@1520591568 after retry RPC@1270225783) -> (continuation of Deferred@398920376 after retry RPC@1817298580) -> (continuation of Deferred@676354679 after retry RPC@1247239311))) after lookup
02:03:50.096 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] connecting to peer
02:03:50.097 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:03:50.097 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] cleaning up while in state CONNECTING due to: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:03:50.097 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:03:50.234 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:03:50.235 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)[L],26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535),4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)]}]
02:03:50.235 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=11, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=318), Traces: [0ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [0ms] delaying RPC due to: Illegal state: connection is terminated, [15ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [17ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [17ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [35ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [38ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [39ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [41ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [41ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [55ms] refreshing cache from master, [55ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [57ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [59ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [60ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [60ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [96ms] refreshing cache from master, [96ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [98ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [100ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [101ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [102ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [175ms] refreshing cache from master, [176ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [178ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [180ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [181ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [181ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [315ms] refreshing cache from master, [315ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [318ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@1357265010(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@3bbebf6f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1520591568 after retry RPC@1270225783) -> (continuation of Deferred@398920376 after retry RPC@1817298580) -> (continuation of Deferred@676354679 after retry RPC@1247239311) -> (continuation of Deferred@233512930 after retry RPC@2108201460), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@276385a0 -> wakeup thread Time-limited test -> (continuation of Deferred@1520591568 after retry RPC@1270225783) -> (continuation of Deferred@398920376 after retry RPC@1817298580) -> (continuation of Deferred@676354679 after retry RPC@1247239311) -> (continuation of Deferred@233512930 after retry RPC@2108201460))) after lookup
02:03:50.236 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] connecting to peer
02:03:50.237 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:03:50.237 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] cleaning up while in state CONNECTING due to: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:03:50.237 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:03:51.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:51.107776 31811 raft_consensus.cc:493] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
02:03:51.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:51.108076 31811 raft_consensus.cc:515] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:03:51.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:51.109383 31811 leader_election.cc:290] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130:35181), 06f0647651a749f8a533afd451fe5fcb (127.29.165.129:45737)
02:03:51.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:51.110124 31687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.165.129:45737: connect: Connection refused (error 111)
02:03:51.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:51.111094 31687 leader_election.cc:336] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 06f0647651a749f8a533afd451fe5fcb (127.29.165.129:45737): Network error: Client connection negotiation failed: client connection to 127.29.165.129:45737: connect: Connection refused (error 111)
02:03:51.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:51.113498 31579 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "830b8d1af48f4d34b281a548f1dbe5ef" candidate_uuid: "26ea15144933497a98a05f83b05e8f93" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" is_pre_election: true
02:03:51.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:51.114076 31686 leader_election.cc:304] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26ea15144933497a98a05f83b05e8f93; no voters: 06f0647651a749f8a533afd451fe5fcb, 4e39ec54c2d642feafa81dc5d4f066c2
02:03:51.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:51.114303 31811 raft_consensus.cc:2749] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
02:03:51.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:51.219983 31816 raft_consensus.cc:493] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 06f0647651a749f8a533afd451fe5fcb)
02:03:51.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:51.220135 31816 raft_consensus.cc:515] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:03:51.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:51.220575 31816 leader_election.cc:290] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 06f0647651a749f8a533afd451fe5fcb (127.29.165.129:45737), 26ea15144933497a98a05f83b05e8f93 (127.29.165.131:44535)
02:03:51.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:51.222008 31514 leader_election.cc:336] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 06f0647651a749f8a533afd451fe5fcb (127.29.165.129:45737): Network error: Client connection negotiation failed: client connection to 127.29.165.129:45737: connect: Connection refused (error 111)
02:03:51.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:51.225471 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "830b8d1af48f4d34b281a548f1dbe5ef" candidate_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "26ea15144933497a98a05f83b05e8f93" is_pre_election: true
02:03:51.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:51.225692 31752 raft_consensus.cc:2468] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e39ec54c2d642feafa81dc5d4f066c2 in term 4.
02:03:51.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:51.226307 31513 leader_election.cc:304] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26ea15144933497a98a05f83b05e8f93, 4e39ec54c2d642feafa81dc5d4f066c2; no voters: 06f0647651a749f8a533afd451fe5fcb
02:03:51.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:51.226490 31816 raft_consensus.cc:2804] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 5 FOLLOWER]: Leader pre-election won for term 6
02:03:51.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:51.226557 31816 raft_consensus.cc:493] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 06f0647651a749f8a533afd451fe5fcb)
02:03:51.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:51.226594 31816 raft_consensus.cc:3060] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 5 FOLLOWER]: Advancing to term 6
02:03:51.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:51.227440 31816 raft_consensus.cc:515] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:03:51.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:51.227810 31816 leader_election.cc:290] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [CANDIDATE]: Term 6 election: Requested vote from peers 06f0647651a749f8a533afd451fe5fcb (127.29.165.129:45737), 26ea15144933497a98a05f83b05e8f93 (127.29.165.131:44535)
02:03:51.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:51.228303 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "830b8d1af48f4d34b281a548f1dbe5ef" candidate_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "26ea15144933497a98a05f83b05e8f93"
02:03:51.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:51.228467 31752 raft_consensus.cc:3060] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 4 FOLLOWER]: Advancing to term 6
02:03:51.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:51.228708 31514 leader_election.cc:336] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 06f0647651a749f8a533afd451fe5fcb (127.29.165.129:45737): Network error: Client connection negotiation failed: client connection to 127.29.165.129:45737: connect: Connection refused (error 111)
02:03:51.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:51.230360 31752 raft_consensus.cc:2468] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e39ec54c2d642feafa81dc5d4f066c2 in term 6.
02:03:51.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:51.230742 31513 leader_election.cc:304] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26ea15144933497a98a05f83b05e8f93, 4e39ec54c2d642feafa81dc5d4f066c2; no voters: 06f0647651a749f8a533afd451fe5fcb
02:03:51.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:51.230926 31816 raft_consensus.cc:2804] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 6 FOLLOWER]: Leader election won for term 6
02:03:51.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:51.231158 31816 raft_consensus.cc:697] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 6 LEADER]: Becoming Leader. State: Replica: 4e39ec54c2d642feafa81dc5d4f066c2, State: Running, Role: LEADER
02:03:51.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:51.231479 31816 consensus_queue.cc:237] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 5.20, Last appended by leader: 20, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:03:51.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:51.234494 30542 catalog_manager.cc:5696] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 reported cstate change: term changed from 5 to 6, leader changed from 06f0647651a749f8a533afd451fe5fcb (127.29.165.129) to 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130). New cstate: current_term: 6 leader_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } health_report { overall_health: UNKNOWN } } }
02:03:51.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:51.768087 31514 consensus_peers.cc:597] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 -> Peer 06f0647651a749f8a533afd451fe5fcb (127.29.165.129:45737): Couldn't send request to peer 06f0647651a749f8a533afd451fe5fcb. Status: Network error: Client connection negotiation failed: client connection to 127.29.165.129:45737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:03:51.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:51.792634 31752 raft_consensus.cc:1275] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 6 FOLLOWER]: Refusing update from remote peer 4e39ec54c2d642feafa81dc5d4f066c2: Log matching property violated. Preceding OpId in replica: term: 4 index: 16. Preceding OpId from leader: term: 6 index: 21. (index mismatch)
02:03:51.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:51.793272 31833 consensus_queue.cc:1048] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 16, Time since last communication: 0.000s
02:03:53.834 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:03:53.835 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737),26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535),4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)[L]]}]
02:03:53.835 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=13, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=3919), Traces: [0ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [0ms] delaying RPC due to: Illegal state: connection is terminated, [15ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [17ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [17ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [35ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [38ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [39ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [41ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [41ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [55ms] refreshing cache from master, [55ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [57ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [59ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [60ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [60ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [96ms] refreshing cache from master, [96ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [98ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [100ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [101ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [102ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [175ms] refreshing cache from master, [176ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [178ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [180ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [181ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [181ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [315ms] refreshing cache from master, [315ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [318ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [319ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [321ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [321ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [3915ms] refreshing cache from master, [3915ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [3918ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@1357265010(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@3bbebf6f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1520591568 after retry RPC@1270225783) -> (continuation of Deferred@398920376 after retry RPC@1817298580) -> (continuation of Deferred@676354679 after retry RPC@1247239311) -> (continuation of Deferred@233512930 after retry RPC@2108201460) -> (continuation of Deferred@1857234601 after retry RPC@1475179954), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@276385a0 -> wakeup thread Time-limited test -> (continuation of Deferred@1520591568 after retry RPC@1270225783) -> (continuation of Deferred@398920376 after retry RPC@1817298580) -> (continuation of Deferred@676354679 after retry RPC@1247239311) -> (continuation of Deferred@233512930 after retry RPC@2108201460) -> (continuation of Deferred@1857234601 after retry RPC@1475179954))) after lookup
02:03:53.836 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] connecting to peer
02:03:53.837 [DEBUG - kudu-nio-6] (Connection.java:757) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] Successfully connected to peer
02:03:53.861 [DEBUG - kudu-nio-6] (Negotiator.java:919) Authenticated connection [id: 0x04bbc6da, L:/127.0.0.1:35618 - R:/127.29.165.130:35181] using TOKEN/null
02:03:53.880 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.29.165.129:45737
02:03:53.882 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

02:03:53.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:53.882817 30358 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskxPorqF/build/debug/bin/kudu
02:03:53.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskxPorqF/build/debug/bin/kudu
02:03:53.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/wal
02:03:53.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/data
02:03:53.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:03:53.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:03:53.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:03:53.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:03:53.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:03:53.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:03:53.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/logs
02:03:53.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/data/info.pb
02:03:53.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:03:53.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:03:53.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:03:53.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:03:53.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:03:53.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:03:53.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:03:53.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:03:53.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:03:53.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:03:53.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.165.129:45737
02:03:53.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.165.129
02:03:53.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.165.190:39217,127.29.165.189:39733,127.29.165.188:38583
02:03:53.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=46663
02:03:53.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.165.129
02:03:53.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.165.148:38003
02:03:53.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:03:53.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:03:53.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:03:53.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:53.948043 31514 consensus_peers.cc:597] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 -> Peer 06f0647651a749f8a533afd451fe5fcb (127.29.165.129:45737): Couldn't send request to peer 06f0647651a749f8a533afd451fe5fcb. Status: Network error: Client connection negotiation failed: client connection to 127.29.165.129:45737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
02:03:53.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:53.988993 31853 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:03:53.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:53.989183 31853 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:03:53.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:53.989226 31853 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:03:53.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:53.992486 31853 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:03:53.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:53.992579 31853 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.165.129
02:03:53.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:53.996225 31853 tablet_server_runner.cc:78] Tablet server non-default flags:
02:03:53.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:03:53.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.165.148:38003
02:03:53.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:03:53.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:03:53.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/data
02:03:53.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/wal
02:03:53.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:03:53.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:03:53.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.165.129:45737
02:03:53.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:03:53.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:03:53.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:03:53.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/data/info.pb
02:03:53.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.165.129
02:03:53.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=46663
02:03:53.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.165.190:39217,127.29.165.189:39733,127.29.165.188:38583
02:03:53.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:03:53.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.31853
02:03:53.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:03:53.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:03:53.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:03:53.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:03:53.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.165.129
02:03:53.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/logs
02:03:53.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:03:53.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:03:53.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:03:53.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:03:53.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:03:53.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
02:03:53.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type DEBUG
02:03:53.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 01:43:15 UTC on bdcb31816ec0
02:03:53.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11948
02:03:53.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:53.997145 31853 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:03:53.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:53.998154 31853 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:03:54.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:54.003928 31859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:54.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:54.003944 31861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:54.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:54.003947 31858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:54.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.004287 31853 server_base.cc:1061] running on GCE node
02:03:54.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.004649 31853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:03:54.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.005174 31853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:03:54.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.006327 31853 hybrid_clock.cc:648] HybridClock initialized: now 1779069834006307 us; error 39 us; skew 500 ppm
02:03:54.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.008030 31853 webserver.cc:533] Webserver started at http://127.29.165.129:46663/ using document root <none> and password file <none>
02:03:54.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.008493 31853 fs_manager.cc:362] Metadata directory not provided
02:03:54.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.008549 31853 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:03:54.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.011562 31853 fs_manager.cc:714] Time spent opening directory manager: real 0.002s	user 0.000s	sys 0.003s
02:03:54.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.013253 31867 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:03:54.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.014158 31853 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.000s	sys 0.002s
02:03:54.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.014288 31853 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/data,/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/wal
02:03:54.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "06f0647651a749f8a533afd451fe5fcb"
02:03:54.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:03:36 on dist-test-slave-838s"
02:03:54.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.014775 31853 fs_report.cc:389] FS layout report
02:03:54.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:03:54.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/wal
02:03:54.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/wal
02:03:54.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/data/data
02:03:54.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:03:54.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:03:54.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:03:54.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:03:54.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:03:54.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:03:54.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:03:54.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:03:54.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:03:54.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:03:54.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:03:54.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.029592 31853 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:03:54.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.030237 31853 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:03:54.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.030431 31853 kserver.cc:163] Server-wide thread pool size limit: 3276
02:03:54.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.031085 31853 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:03:54.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.032289 31874 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
02:03:54.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.035753 31853 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:03:54.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.035827 31853 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.004s	user 0.000s	sys 0.000s
02:03:54.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.035889 31853 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:03:54.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.037338 31853 ts_tablet_manager.cc:616] Registered 1 tablets
02:03:54.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.037405 31853 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s	user 0.000s	sys 0.000s
02:03:54.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.037671 31874 tablet_bootstrap.cc:492] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb: Bootstrap starting.
02:03:54.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.048205 31853 rpc_server.cc:307] RPC server started. Bound to: 127.29.165.129:45737
02:03:54.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.048221 31981 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.165.129:45737 every 8 connection(s)
02:03:54.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.049155 31853 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/data/info.pb
02:03:54.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.050175 30358 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskxPorqF/build/debug/bin/kudu as pid 31853
02:03:54.051 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:03:54.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.054906 31874 log.cc:826] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb: Log is configured to *not* fsync() on all Append() calls
02:03:54.060 [DEBUG - kudu-nio-6] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 18 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956, tablet=830b8d1af48f4d34b281a548f1dbe5ef, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:03:54.061 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.29.165.130:35181
02:03:54.061 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

02:03:54.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.061606 30358 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskxPorqF/build/debug/bin/kudu with pid 31496
02:03:54.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.064554 31982 heartbeater.cc:344] Connected to a master server at 127.29.165.190:39217
02:03:54.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.064561 31984 heartbeater.cc:344] Connected to a master server at 127.29.165.189:39733
02:03:54.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.064836 31982 heartbeater.cc:461] Registering TS with master...
02:03:54.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.064836 31984 heartbeater.cc:461] Registering TS with master...
02:03:54.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.065418 31982 heartbeater.cc:507] Master 127.29.165.190:39217 requested a full tablet report, sending...
02:03:54.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.065418 31984 heartbeater.cc:507] Master 127.29.165.189:39733 requested a full tablet report, sending...
02:03:54.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.067409 30472 ts_manager.cc:194] Re-registered known tserver with Master: 06f0647651a749f8a533afd451fe5fcb (127.29.165.129:45737)
02:03:54.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.068106 30399 ts_manager.cc:194] Re-registered known tserver with Master: 06f0647651a749f8a533afd451fe5fcb (127.29.165.129:45737)
02:03:54.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.069314 31983 heartbeater.cc:344] Connected to a master server at 127.29.165.188:38583
02:03:54.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.069399 31983 heartbeater.cc:461] Registering TS with master...
02:03:54.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.069602 31983 heartbeater.cc:507] Master 127.29.165.188:38583 requested a full tablet report, sending...
02:03:54.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.070529 30543 ts_manager.cc:194] Re-registered known tserver with Master: 06f0647651a749f8a533afd451fe5fcb (127.29.165.129:45737)
02:03:54.071 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:03:54.071 [DEBUG - kudu-nio-6] (Connection.java:239) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] handling channelInactive
02:03:54.071 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] cleaning up while in state READY due to: connection closed
02:03:54.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.071604 30543 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.165.129:38773
02:03:54.072 [DEBUG - Time-limited test] (Connection.java:748) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] connecting to peer
02:03:54.073 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:03:54.073 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] cleaning up while in state CONNECTING due to: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:03:54.073 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:03:54.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.076480 31874 tablet_bootstrap.cc:492] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=20 ignored=0} inserts{seen=15 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:03:54.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.076809 31874 tablet_bootstrap.cc:492] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb: Bootstrap complete.
02:03:54.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.077777 31874 ts_tablet_manager.cc:1403] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb: Time spent bootstrapping tablet: real 0.040s	user 0.028s	sys 0.004s
02:03:54.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.080034 31874 raft_consensus.cc:359] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:03:54.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.080379 31874 raft_consensus.cc:740] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06f0647651a749f8a533afd451fe5fcb, State: Initialized, Role: FOLLOWER
02:03:54.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.080754 31874 consensus_queue.cc:260] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 20, Last appended: 5.20, Last appended by leader: 20, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:03:54.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.081388 31983 heartbeater.cc:499] Master 127.29.165.188:38583 was elected leader, sending a full tablet report...
02:03:54.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:54.081612 31874 ts_tablet_manager.cc:1434] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb: Time spent starting tablet: real 0.004s	user 0.006s	sys 0.000s
02:03:54.094 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:03:54.094 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737),26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535),4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)[L]]}]
02:03:54.094 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=2, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=23), Traces: [0ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [1ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [1ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [19ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [22ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@78751379(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@38cd1352 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@29876534 -> wakeup thread Time-limited test)) after lookup
02:03:54.095 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] connecting to peer
02:03:54.096 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:03:54.096 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] cleaning up while in state CONNECTING due to: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:03:54.096 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:03:54.113 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:03:54.114 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737),26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535),4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)[L]]}]
02:03:54.114 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=4, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=42), Traces: [0ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [1ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [1ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [19ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [22ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [23ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [24ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [24ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [41ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@78751379(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@38cd1352 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@20774725 after retry RPC@318805139), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@29876534 -> wakeup thread Time-limited test -> (continuation of Deferred@20774725 after retry RPC@318805139))) after lookup
02:03:54.115 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] connecting to peer
02:03:54.116 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:03:54.116 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] cleaning up while in state CONNECTING due to: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:03:54.116 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:03:54.134 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:03:54.134 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737),26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535),4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)[L]]}]
02:03:54.134 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=6, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=63), Traces: [0ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [1ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [1ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [19ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [22ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [23ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [24ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [24ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [41ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [42ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [44ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [44ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [59ms] refreshing cache from master, [60ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [61ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@78751379(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@38cd1352 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@20774725 after retry RPC@318805139) -> (continuation of Deferred@642023390 after retry RPC@1983640956), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@29876534 -> wakeup thread Time-limited test -> (continuation of Deferred@20774725 after retry RPC@318805139) -> (continuation of Deferred@642023390 after retry RPC@1983640956))) after lookup
02:03:54.135 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] connecting to peer
02:03:54.136 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:03:54.136 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] cleaning up while in state CONNECTING due to: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:03:54.136 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:03:54.214 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:03:54.214 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737),26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535),4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)[L]]}]
02:03:54.215 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=8, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=143), Traces: [0ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [1ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [1ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [19ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [22ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [23ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [24ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [24ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [41ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [42ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [44ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [44ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [59ms] refreshing cache from master, [60ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [61ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [63ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [64ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [64ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [139ms] refreshing cache from master, [139ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [142ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@78751379(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@38cd1352 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@20774725 after retry RPC@318805139) -> (continuation of Deferred@642023390 after retry RPC@1983640956) -> (continuation of Deferred@2036347979 after retry RPC@1348054542), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@29876534 -> wakeup thread Time-limited test -> (continuation of Deferred@20774725 after retry RPC@318805139) -> (continuation of Deferred@642023390 after retry RPC@1983640956) -> (continuation of Deferred@2036347979 after retry RPC@1348054542))) after lookup
02:03:54.216 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] connecting to peer
02:03:54.217 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:03:54.217 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] cleaning up while in state CONNECTING due to: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:03:54.217 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:03:54.694 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:03:54.694 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737),26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535),4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)[L]]}]
02:03:54.694 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=10, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=623), Traces: [0ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [1ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [1ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [19ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [22ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [23ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [24ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [24ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [41ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [42ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [44ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [44ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [59ms] refreshing cache from master, [60ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [61ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [63ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [64ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [64ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [139ms] refreshing cache from master, [139ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [142ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [144ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [145ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [145ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [619ms] refreshing cache from master, [619ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [621ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@78751379(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@38cd1352 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@20774725 after retry RPC@318805139) -> (continuation of Deferred@642023390 after retry RPC@1983640956) -> (continuation of Deferred@2036347979 after retry RPC@1348054542) -> (continuation of Deferred@1124429744 after retry RPC@44156754), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@29876534 -> wakeup thread Time-limited test -> (continuation of Deferred@20774725 after retry RPC@318805139) -> (continuation of Deferred@642023390 after retry RPC@1983640956) -> (continuation of Deferred@2036347979 after retry RPC@1348054542) -> (continuation of Deferred@1124429744 after retry RPC@44156754))) after lookup
02:03:54.696 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] connecting to peer
02:03:54.697 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:03:54.697 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] cleaning up while in state CONNECTING due to: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:03:54.697 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:03:54.974 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:03:54.975 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737),26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535),4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)[L]]}]
02:03:54.975 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=12, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=903), Traces: [0ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [1ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [1ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [19ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [22ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [23ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [24ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [24ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [41ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [42ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [44ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [44ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [59ms] refreshing cache from master, [60ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [61ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [63ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [64ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [64ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [139ms] refreshing cache from master, [139ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [142ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [144ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [145ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [145ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [619ms] refreshing cache from master, [619ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [621ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [623ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [625ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [625ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [899ms] refreshing cache from master, [899ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [902ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@78751379(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@38cd1352 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@20774725 after retry RPC@318805139) -> (continuation of Deferred@642023390 after retry RPC@1983640956) -> (continuation of Deferred@2036347979 after retry RPC@1348054542) -> (continuation of Deferred@1124429744 after retry RPC@44156754) -> (continuation of Deferred@1357348165 after retry RPC@1559280476), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@29876534 -> wakeup thread Time-limited test -> (continuation of Deferred@20774725 after retry RPC@318805139) -> (continuation of Deferred@642023390 after retry RPC@1983640956) -> (continuation of Deferred@2036347979 after retry RPC@1348054542) -> (continuation of Deferred@1124429744 after retry RPC@44156754) -> (continuation of Deferred@1357348165 after retry RPC@1559280476))) after lookup
02:03:54.976 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] connecting to peer
02:03:54.977 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:03:54.977 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] cleaning up while in state CONNECTING due to: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:03:54.977 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:03:55.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:55.380169 31995 raft_consensus.cc:493] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4e39ec54c2d642feafa81dc5d4f066c2)
02:03:55.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:55.380304 31995 raft_consensus.cc:515] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:03:55.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:55.380782 31995 leader_election.cc:290] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130:35181), 06f0647651a749f8a533afd451fe5fcb (127.29.165.129:45737)
02:03:55.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:55.382095 31686 leader_election.cc:336] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130:35181): Network error: Client connection negotiation failed: client connection to 127.29.165.130:35181: connect: Connection refused (error 111)
02:03:55.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:55.385780 31936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "830b8d1af48f4d34b281a548f1dbe5ef" candidate_uuid: "26ea15144933497a98a05f83b05e8f93" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "06f0647651a749f8a533afd451fe5fcb" is_pre_election: true
02:03:55.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:55.386060 31936 raft_consensus.cc:2468] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26ea15144933497a98a05f83b05e8f93 in term 5.
02:03:55.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:55.386688 31687 leader_election.cc:304] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06f0647651a749f8a533afd451fe5fcb, 26ea15144933497a98a05f83b05e8f93; no voters: 4e39ec54c2d642feafa81dc5d4f066c2
02:03:55.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:55.386896 31995 raft_consensus.cc:2804] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 6 FOLLOWER]: Leader pre-election won for term 7
02:03:55.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:55.386979 31995 raft_consensus.cc:493] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4e39ec54c2d642feafa81dc5d4f066c2)
02:03:55.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:55.387030 31995 raft_consensus.cc:3060] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 6 FOLLOWER]: Advancing to term 7
02:03:55.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:55.387879 31995 raft_consensus.cc:515] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:03:55.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:55.388300 31995 leader_election.cc:290] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [CANDIDATE]: Term 7 election: Requested vote from peers 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130:35181), 06f0647651a749f8a533afd451fe5fcb (127.29.165.129:45737)
02:03:55.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:55.388674 31936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "830b8d1af48f4d34b281a548f1dbe5ef" candidate_uuid: "26ea15144933497a98a05f83b05e8f93" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "06f0647651a749f8a533afd451fe5fcb"
02:03:55.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:55.388803 31936 raft_consensus.cc:3060] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 5 FOLLOWER]: Advancing to term 7
02:03:55.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:55.389882 31686 leader_election.cc:336] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130:35181): Network error: Client connection negotiation failed: client connection to 127.29.165.130:35181: connect: Connection refused (error 111)
02:03:55.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:55.390841 31936 raft_consensus.cc:2468] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26ea15144933497a98a05f83b05e8f93 in term 7.
02:03:55.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:55.391151 31687 leader_election.cc:304] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06f0647651a749f8a533afd451fe5fcb, 26ea15144933497a98a05f83b05e8f93; no voters: 4e39ec54c2d642feafa81dc5d4f066c2
02:03:55.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:55.391346 31995 raft_consensus.cc:2804] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 7 FOLLOWER]: Leader election won for term 7
02:03:55.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:55.391573 31995 raft_consensus.cc:697] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 7 LEADER]: Becoming Leader. State: Replica: 26ea15144933497a98a05f83b05e8f93, State: Running, Role: LEADER
02:03:55.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:55.391899 31995 consensus_queue.cc:237] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 6.24, Last appended by leader: 24, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:03:55.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:55.394377 30543 catalog_manager.cc:5696] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 reported cstate change: term changed from 6 to 7, leader changed from 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130) to 26ea15144933497a98a05f83b05e8f93 (127.29.165.131). New cstate: current_term: 7 leader_uuid: "26ea15144933497a98a05f83b05e8f93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } health_report { overall_health: HEALTHY } } }
02:03:55.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:55.884065 31936 raft_consensus.cc:1275] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 7 FOLLOWER]: Refusing update from remote peer 26ea15144933497a98a05f83b05e8f93: Log matching property violated. Preceding OpId in replica: term: 5 index: 20. Preceding OpId from leader: term: 7 index: 25. (index mismatch)
02:03:55.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:55.884819 31995 consensus_queue.cc:1048] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 20, Time since last communication: 0.000s
02:03:55.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:55.894956 31686 consensus_peers.cc:597] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 -> Peer 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130:35181): Couldn't send request to peer 4e39ec54c2d642feafa81dc5d4f066c2. Status: Network error: Client connection negotiation failed: client connection to 127.29.165.130:35181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:03:56.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:56.331249 31686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.165.130:35181: connect: Connection refused (error 111) [suppressed 3 similar messages]
02:03:58.374 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:03:58.374 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737),26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)[L],4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)]}]
02:03:58.375 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=14, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=4303), Traces: [0ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [1ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [1ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [19ms] refreshing cache from master, [20ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [22ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [23ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [24ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [24ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [41ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [42ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [44ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [44ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [59ms] refreshing cache from master, [60ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [61ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [63ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [64ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [64ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [139ms] refreshing cache from master, [139ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [142ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [144ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [145ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [145ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [619ms] refreshing cache from master, [619ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [621ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [623ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [625ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [625ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [899ms] refreshing cache from master, [899ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [902ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [903ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [905ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [905ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [4299ms] refreshing cache from master, [4299ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [4301ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@78751379(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@38cd1352 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@20774725 after retry RPC@318805139) -> (continuation of Deferred@642023390 after retry RPC@1983640956) -> (continuation of Deferred@2036347979 after retry RPC@1348054542) -> (continuation of Deferred@1124429744 after retry RPC@44156754) -> (continuation of Deferred@1357348165 after retry RPC@1559280476) -> (continuation of Deferred@923757392 after retry RPC@1853623478), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@29876534 -> wakeup thread Time-limited test -> (continuation of Deferred@20774725 after retry RPC@318805139) -> (continuation of Deferred@642023390 after retry RPC@1983640956) -> (continuation of Deferred@2036347979 after retry RPC@1348054542) -> (continuation of Deferred@1124429744 after retry RPC@44156754) -> (continuation of Deferred@1357348165 after retry RPC@1559280476) -> (continuation of Deferred@923757392 after retry RPC@1853623478))) after lookup
02:03:58.376 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] connecting to peer
02:03:58.377 [DEBUG - kudu-nio-6] (Connection.java:757) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] Successfully connected to peer
02:03:58.394 [DEBUG - kudu-nio-6] (Negotiator.java:919) Authenticated connection [id: 0x95a4d30f, L:/127.0.0.1:59154 - R:/127.29.165.131:44535] using TOKEN/null
02:03:58.411 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.29.165.130:35181
02:03:58.412 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

02:03:58.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.412602 30358 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskxPorqF/build/debug/bin/kudu
02:03:58.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskxPorqF/build/debug/bin/kudu
02:03:58.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/wal
02:03:58.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/data
02:03:58.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:03:58.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:03:58.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:03:58.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:03:58.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:03:58.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:03:58.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/logs
02:03:58.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/data/info.pb
02:03:58.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:03:58.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:03:58.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:03:58.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:03:58.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:03:58.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:03:58.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:03:58.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:03:58.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:03:58.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:03:58.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.165.130:35181
02:03:58.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.165.130
02:03:58.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.165.190:39217,127.29.165.189:39733,127.29.165.188:38583
02:03:58.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=33955
02:03:58.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.165.130
02:03:58.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.165.148:38003
02:03:58.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:03:58.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:03:58.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:03:58.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:58.430166 31686 consensus_peers.cc:597] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 -> Peer 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130:35181): Couldn't send request to peer 4e39ec54c2d642feafa81dc5d4f066c2. Status: Network error: Client connection negotiation failed: client connection to 127.29.165.130:35181: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
02:03:58.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:58.517030 32039 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:03:58.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:58.517316 32039 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:03:58.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:58.517364 32039 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:03:58.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:58.520686 32039 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:03:58.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:58.520793 32039 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.165.130
02:03:58.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.524659 32039 tablet_server_runner.cc:78] Tablet server non-default flags:
02:03:58.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:03:58.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.165.148:38003
02:03:58.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:03:58.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:03:58.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/data
02:03:58.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/wal
02:03:58.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:03:58.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:03:58.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.165.130:35181
02:03:58.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:03:58.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:03:58.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:03:58.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/data/info.pb
02:03:58.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.165.130
02:03:58.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=33955
02:03:58.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.165.190:39217,127.29.165.189:39733,127.29.165.188:38583
02:03:58.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:03:58.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.32039
02:03:58.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:03:58.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:03:58.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:03:58.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:03:58.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.165.130
02:03:58.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/logs
02:03:58.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:03:58.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:03:58.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:03:58.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:03:58.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:03:58.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
02:03:58.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type DEBUG
02:03:58.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 01:43:15 UTC on bdcb31816ec0
02:03:58.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11948
02:03:58.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.525730 32039 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:03:58.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.526858 32039 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:03:58.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:58.533226 32047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:58.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:58.533207 32045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:58.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:58.533211 32044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:03:58.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.533555 32039 server_base.cc:1061] running on GCE node
02:03:58.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.533967 32039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:03:58.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.534535 32039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:03:58.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.535713 32039 hybrid_clock.cc:648] HybridClock initialized: now 1779069838535680 us; error 46 us; skew 500 ppm
02:03:58.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.539258 32039 webserver.cc:533] Webserver started at http://127.29.165.130:33955/ using document root <none> and password file <none>
02:03:58.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.539863 32039 fs_manager.cc:362] Metadata directory not provided
02:03:58.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.539950 32039 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:03:58.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.543358 32039 fs_manager.cc:714] Time spent opening directory manager: real 0.002s	user 0.003s	sys 0.000s
02:03:58.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.545248 32053 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:03:58.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.546175 32039 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
02:03:58.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.546299 32039 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/data,/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/wal
02:03:58.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "4e39ec54c2d642feafa81dc5d4f066c2"
02:03:58.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:03:36 on dist-test-slave-838s"
02:03:58.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.546741 32039 fs_report.cc:389] FS layout report
02:03:58.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:03:58.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/wal
02:03:58.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/wal
02:03:58.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/data/data
02:03:58.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:03:58.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:03:58.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:03:58.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:03:58.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:03:58.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:03:58.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:03:58.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:03:58.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:03:58.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:03:58.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:03:58.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.556188 32039 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:03:58.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.556865 32039 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:03:58.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.557107 32039 kserver.cc:163] Server-wide thread pool size limit: 3276
02:03:58.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.557722 32039 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:03:58.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.559176 32060 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
02:03:58.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.562408 32039 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:03:58.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.562490 32039 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.004s	user 0.000s	sys 0.000s
02:03:58.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.562556 32039 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:03:58.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.564118 32039 ts_tablet_manager.cc:616] Registered 1 tablets
02:03:58.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.564189 32039 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s	user 0.000s	sys 0.000s
02:03:58.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.564559 32060 tablet_bootstrap.cc:492] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2: Bootstrap starting.
02:03:58.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.575331 32039 rpc_server.cc:307] RPC server started. Bound to: 127.29.165.130:35181
02:03:58.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.575387 32167 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.165.130:35181 every 8 connection(s)
02:03:58.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.576328 32039 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/data/info.pb
02:03:58.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.578213 30358 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskxPorqF/build/debug/bin/kudu as pid 32039
02:03:58.579 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:03:58.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.585567 32060 log.cc:826] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2: Log is configured to *not* fsync() on all Append() calls
02:03:58.588 [DEBUG - kudu-nio-6] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 21 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956, tablet=830b8d1af48f4d34b281a548f1dbe5ef, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:03:58.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.588693 32170 heartbeater.cc:344] Connected to a master server at 127.29.165.189:39733
02:03:58.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.588922 32170 heartbeater.cc:461] Registering TS with master...
02:03:58.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.589475 32170 heartbeater.cc:507] Master 127.29.165.189:39733 requested a full tablet report, sending...
02:03:58.591 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.29.165.131:44535
02:03:58.591 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

02:03:58.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.591671 30358 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskxPorqF/build/debug/bin/kudu with pid 31669
02:03:58.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.591730 32169 heartbeater.cc:344] Connected to a master server at 127.29.165.188:38583
02:03:58.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.591774 30472 ts_manager.cc:194] Re-registered known tserver with Master: 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130:35181)
02:03:58.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.591807 32169 heartbeater.cc:461] Registering TS with master...
02:03:58.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.592036 32169 heartbeater.cc:507] Master 127.29.165.188:38583 requested a full tablet report, sending...
02:03:58.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.592923 30543 ts_manager.cc:194] Re-registered known tserver with Master: 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130:35181)
02:03:58.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.593513 32168 heartbeater.cc:344] Connected to a master server at 127.29.165.190:39217
02:03:58.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.593601 32168 heartbeater.cc:461] Registering TS with master...
02:03:58.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.593806 32168 heartbeater.cc:507] Master 127.29.165.190:39217 requested a full tablet report, sending...
02:03:58.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.593957 30543 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.165.130:43685
02:03:58.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.594730 30400 ts_manager.cc:194] Re-registered known tserver with Master: 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130:35181)
02:03:58.599 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:03:58.599 [DEBUG - kudu-nio-6] (Connection.java:239) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] handling channelInactive
02:03:58.599 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] cleaning up while in state READY due to: connection closed
02:03:58.600 [DEBUG - Time-limited test] (Connection.java:748) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] connecting to peer
02:03:58.600 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:58.601 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] cleaning up while in state CONNECTING due to: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:58.601 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:58.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.606446 32060 tablet_bootstrap.cc:492] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=24 overwritten=0 applied=24 ignored=0} inserts{seen=18 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:03:58.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.606916 32060 tablet_bootstrap.cc:492] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2: Bootstrap complete.
02:03:58.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.608047 32060 ts_tablet_manager.cc:1403] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2: Time spent bootstrapping tablet: real 0.044s	user 0.026s	sys 0.009s
02:03:58.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.610083 32060 raft_consensus.cc:359] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:03:58.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.610395 32060 raft_consensus.cc:740] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e39ec54c2d642feafa81dc5d4f066c2, State: Initialized, Role: FOLLOWER
02:03:58.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.611032 32060 consensus_queue.cc:260] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 6.24, Last appended by leader: 24, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:03:58.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.611999 32060 ts_tablet_manager.cc:1434] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2: Time spent starting tablet: real 0.004s	user 0.004s	sys 0.000s
02:03:58.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:58.612385 32169 heartbeater.cc:499] Master 127.29.165.188:38583 was elected leader, sending a full tablet report...
02:03:58.614 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:03:58.614 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737),26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)[L],4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)]}]
02:03:58.614 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=2, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=15), Traces: [0ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [2ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [2ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [14ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@1668648227(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@6d623447 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@65495417 -> wakeup thread Time-limited test)) after lookup
02:03:58.615 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] connecting to peer
02:03:58.616 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:58.616 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] cleaning up while in state CONNECTING due to: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:58.616 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:58.635 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:03:58.635 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737),26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)[L],4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)]}]
02:03:58.635 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=4, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=36), Traces: [0ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [2ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [2ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [14ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [16ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [17ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [17ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [32ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [35ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@1668648227(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@6d623447 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@406275538 after retry RPC@930169647), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@65495417 -> wakeup thread Time-limited test -> (continuation of Deferred@406275538 after retry RPC@930169647))) after lookup
02:03:58.636 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] connecting to peer
02:03:58.637 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:58.637 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] cleaning up while in state CONNECTING due to: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:58.637 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:58.674 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:03:58.675 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737),26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)[L],4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)]}]
02:03:58.675 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=6, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=75), Traces: [0ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [2ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [2ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [14ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [16ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [17ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [17ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [32ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [35ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [37ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [38ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [38ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [72ms] refreshing cache from master, [72ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [75ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@1668648227(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@6d623447 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@406275538 after retry RPC@930169647) -> (continuation of Deferred@1146834109 after retry RPC@485223192), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@65495417 -> wakeup thread Time-limited test -> (continuation of Deferred@406275538 after retry RPC@930169647) -> (continuation of Deferred@1146834109 after retry RPC@485223192))) after lookup
02:03:58.676 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] connecting to peer
02:03:58.677 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:58.677 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] cleaning up while in state CONNECTING due to: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:58.677 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:58.694 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:03:58.694 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737),26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)[L],4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)]}]
02:03:58.695 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=8, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=95), Traces: [0ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [2ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [2ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [14ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [16ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [17ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [17ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [32ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [35ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [37ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [38ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [38ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [72ms] refreshing cache from master, [72ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [75ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [76ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [78ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [78ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [92ms] refreshing cache from master, [92ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [95ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@1668648227(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@6d623447 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@406275538 after retry RPC@930169647) -> (continuation of Deferred@1146834109 after retry RPC@485223192) -> (continuation of Deferred@366423139 after retry RPC@1453827818), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@65495417 -> wakeup thread Time-limited test -> (continuation of Deferred@406275538 after retry RPC@930169647) -> (continuation of Deferred@1146834109 after retry RPC@485223192) -> (continuation of Deferred@366423139 after retry RPC@1453827818))) after lookup
02:03:58.696 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] connecting to peer
02:03:58.696 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:58.697 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] cleaning up while in state CONNECTING due to: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:58.697 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:59.094 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:03:59.095 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737),26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)[L],4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)]}]
02:03:59.095 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=10, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=495), Traces: [0ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [2ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [2ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [14ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [16ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [17ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [17ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [32ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [35ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [37ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [38ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [38ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [72ms] refreshing cache from master, [72ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [75ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [76ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [78ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [78ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [92ms] refreshing cache from master, [92ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [95ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [96ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [98ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [98ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [492ms] refreshing cache from master, [492ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [495ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@1668648227(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@6d623447 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@406275538 after retry RPC@930169647) -> (continuation of Deferred@1146834109 after retry RPC@485223192) -> (continuation of Deferred@366423139 after retry RPC@1453827818) -> (continuation of Deferred@874531923 after retry RPC@1608199371), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@65495417 -> wakeup thread Time-limited test -> (continuation of Deferred@406275538 after retry RPC@930169647) -> (continuation of Deferred@1146834109 after retry RPC@485223192) -> (continuation of Deferred@366423139 after retry RPC@1453827818) -> (continuation of Deferred@874531923 after retry RPC@1608199371))) after lookup
02:03:59.096 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] connecting to peer
02:03:59.097 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:59.097 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] cleaning up while in state CONNECTING due to: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:59.097 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:03:59.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:59.911934 32181 raft_consensus.cc:493] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 26ea15144933497a98a05f83b05e8f93)
02:03:59.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:59.912091 32181 raft_consensus.cc:515] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:03:59.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:59.913129 32181 leader_election.cc:290] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130:35181), 26ea15144933497a98a05f83b05e8f93 (127.29.165.131:44535)
02:03:59.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:59.914031 31870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.165.131:44535: connect: Connection refused (error 111)
02:03:59.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:59.914929 31870 leader_election.cc:336] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 26ea15144933497a98a05f83b05e8f93 (127.29.165.131:44535): Network error: Client connection negotiation failed: client connection to 127.29.165.131:44535: connect: Connection refused (error 111)
02:03:59.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:59.917917 32122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "830b8d1af48f4d34b281a548f1dbe5ef" candidate_uuid: "06f0647651a749f8a533afd451fe5fcb" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" is_pre_election: true
02:03:59.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:59.918262 32122 raft_consensus.cc:2468] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06f0647651a749f8a533afd451fe5fcb in term 6.
02:03:59.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:59.919075 31870 leader_election.cc:304] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06f0647651a749f8a533afd451fe5fcb, 4e39ec54c2d642feafa81dc5d4f066c2; no voters: 26ea15144933497a98a05f83b05e8f93
02:03:59.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:59.919274 32181 raft_consensus.cc:2804] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 7 FOLLOWER]: Leader pre-election won for term 8
02:03:59.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:59.919334 32181 raft_consensus.cc:493] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 7 FOLLOWER]: Starting leader election (detected failure of leader 26ea15144933497a98a05f83b05e8f93)
02:03:59.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:59.919373 32181 raft_consensus.cc:3060] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 7 FOLLOWER]: Advancing to term 8
02:03:59.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:59.920253 32181 raft_consensus.cc:515] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:03:59.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:59.920573 32181 leader_election.cc:290] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [CANDIDATE]: Term 8 election: Requested vote from peers 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130:35181), 26ea15144933497a98a05f83b05e8f93 (127.29.165.131:44535)
02:03:59.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:59.920897 32122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "830b8d1af48f4d34b281a548f1dbe5ef" candidate_uuid: "06f0647651a749f8a533afd451fe5fcb" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "4e39ec54c2d642feafa81dc5d4f066c2"
02:03:59.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:59.921072 32122 raft_consensus.cc:3060] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 6 FOLLOWER]: Advancing to term 8
02:03:59.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:03:59.921516 31870 leader_election.cc:336] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 26ea15144933497a98a05f83b05e8f93 (127.29.165.131:44535): Network error: Client connection negotiation failed: client connection to 127.29.165.131:44535: connect: Connection refused (error 111)
02:03:59.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:59.922859 32122 raft_consensus.cc:2468] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06f0647651a749f8a533afd451fe5fcb in term 8.
02:03:59.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:59.923169 31870 leader_election.cc:304] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06f0647651a749f8a533afd451fe5fcb, 4e39ec54c2d642feafa81dc5d4f066c2; no voters: 26ea15144933497a98a05f83b05e8f93
02:03:59.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:59.923328 32181 raft_consensus.cc:2804] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 8 FOLLOWER]: Leader election won for term 8
02:03:59.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:59.923547 32181 raft_consensus.cc:697] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 8 LEADER]: Becoming Leader. State: Replica: 06f0647651a749f8a533afd451fe5fcb, State: Running, Role: LEADER
02:03:59.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:59.923857 32181 consensus_queue.cc:237] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 7.28, Last appended by leader: 28, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:03:59.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:03:59.926381 30542 catalog_manager.cc:5696] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb reported cstate change: term changed from 7 to 8, leader changed from 26ea15144933497a98a05f83b05e8f93 (127.29.165.131) to 06f0647651a749f8a533afd451fe5fcb (127.29.165.129). New cstate: current_term: 8 leader_uuid: "06f0647651a749f8a533afd451fe5fcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } health_report { overall_health: UNKNOWN } } }
02:04:00.115 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:04:00.117 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)[L],26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535),4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)]}]
02:04:00.117 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=12, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=1517), Traces: [0ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [2ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [2ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [14ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [16ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [17ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [17ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [32ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [35ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [37ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [38ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [38ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [72ms] refreshing cache from master, [72ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [75ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [76ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [78ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [78ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [92ms] refreshing cache from master, [92ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [95ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [96ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [98ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [98ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [492ms] refreshing cache from master, [492ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [495ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [496ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [498ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [498ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [1512ms] refreshing cache from master, [1513ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [1515ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@1668648227(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@6d623447 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@406275538 after retry RPC@930169647) -> (continuation of Deferred@1146834109 after retry RPC@485223192) -> (continuation of Deferred@366423139 after retry RPC@1453827818) -> (continuation of Deferred@874531923 after retry RPC@1608199371) -> (continuation of Deferred@297838778 after retry RPC@402350909), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@65495417 -> wakeup thread Time-limited test -> (continuation of Deferred@406275538 after retry RPC@930169647) -> (continuation of Deferred@1146834109 after retry RPC@485223192) -> (continuation of Deferred@366423139 after retry RPC@1453827818) -> (continuation of Deferred@874531923 after retry RPC@1608199371) -> (continuation of Deferred@297838778 after retry RPC@402350909))) after lookup
02:04:00.118 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] connecting to peer
02:04:00.119 [DEBUG - kudu-nio-5] (Connection.java:757) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] Successfully connected to peer
02:04:00.141 [DEBUG - kudu-nio-5] (Negotiator.java:919) Authenticated connection [id: 0x19d7537b, L:/127.0.0.1:41126 - R:/127.29.165.129:45737] using TOKEN/null
02:04:00.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:04:00.146096 31870 consensus_peers.cc:597] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb -> Peer 26ea15144933497a98a05f83b05e8f93 (127.29.165.131:44535): Couldn't send request to peer 26ea15144933497a98a05f83b05e8f93. Status: Network error: Client connection negotiation failed: client connection to 127.29.165.131:44535: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:04:00.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.146246 32122 raft_consensus.cc:1275] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 8 FOLLOWER]: Refusing update from remote peer 06f0647651a749f8a533afd451fe5fcb: Log matching property violated. Preceding OpId in replica: term: 6 index: 24. Preceding OpId from leader: term: 8 index: 30. (index mismatch)
02:04:00.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.146868 32181 consensus_queue.cc:1048] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 24, Time since last communication: 0.000s
02:04:00.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.153743 32201 mvcc.cc:204] Tried to move back new op lower bound from 7287070065230774272 to 7287070064331001856. Current Snapshot: MvccSnapshot[applied={T|T < 7287070065230774272}]
02:04:00.166 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.29.165.131:44535
02:04:00.166 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

02:04:00.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.166836 30358 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskxPorqF/build/debug/bin/kudu
02:04:00.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskxPorqF/build/debug/bin/kudu
02:04:00.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/wal
02:04:00.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/data
02:04:00.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:04:00.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:04:00.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:04:00.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:04:00.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:04:00.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:04:00.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/logs
02:04:00.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/data/info.pb
02:04:00.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:04:00.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:04:00.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:04:00.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:04:00.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:04:00.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:04:00.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:04:00.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:04:00.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:04:00.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:04:00.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.165.131:44535
02:04:00.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.165.131
02:04:00.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.165.190:39217,127.29.165.189:39733,127.29.165.188:38583
02:04:00.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45365
02:04:00.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.165.131
02:04:00.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.165.148:38003
02:04:00.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:04:00.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:04:00.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:04:00.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:04:00.269634 32208 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:04:00.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:04:00.269870 32208 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:04:00.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:04:00.269930 32208 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:04:00.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:04:00.273422 32208 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:04:00.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:04:00.273573 32208 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.165.131
02:04:00.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.277457 32208 tablet_server_runner.cc:78] Tablet server non-default flags:
02:04:00.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:04:00.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.165.148:38003
02:04:00.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:04:00.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:04:00.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/data
02:04:00.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/wal
02:04:00.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:04:00.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:04:00.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.165.131:44535
02:04:00.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:04:00.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:04:00.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:04:00.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/data/info.pb
02:04:00.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.165.131
02:04:00.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45365
02:04:00.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.165.190:39217,127.29.165.189:39733,127.29.165.188:38583
02:04:00.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:04:00.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.32208
02:04:00.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:04:00.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:04:00.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:04:00.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:04:00.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.165.131
02:04:00.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/logs
02:04:00.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:04:00.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:04:00.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:04:00.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:04:00.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:04:00.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
02:04:00.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type DEBUG
02:04:00.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 01:43:15 UTC on bdcb31816ec0
02:04:00.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11948
02:04:00.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.278878 32208 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:04:00.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.280093 32208 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:04:00.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:04:00.286528 32213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:04:00.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:04:00.286505 32214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:04:00.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:04:00.286509 32216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:04:00.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.287161 32208 server_base.cc:1061] running on GCE node
02:04:00.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.287508 32208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:04:00.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.287976 32208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:04:00.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.289186 32208 hybrid_clock.cc:648] HybridClock initialized: now 1779069840289168 us; error 31 us; skew 500 ppm
02:04:00.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.291175 32208 webserver.cc:533] Webserver started at http://127.29.165.131:45365/ using document root <none> and password file <none>
02:04:00.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.291776 32208 fs_manager.cc:362] Metadata directory not provided
02:04:00.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.291860 32208 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:04:00.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.295132 32208 fs_manager.cc:714] Time spent opening directory manager: real 0.002s	user 0.003s	sys 0.000s
02:04:00.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.296994 32222 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:04:00.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.297971 32208 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
02:04:00.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.298100 32208 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/data,/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/wal
02:04:00.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "26ea15144933497a98a05f83b05e8f93"
02:04:00.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:03:36 on dist-test-slave-838s"
02:04:00.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.298516 32208 fs_report.cc:389] FS layout report
02:04:00.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:04:00.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/wal
02:04:00.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/wal
02:04:00.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/data/data
02:04:00.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:04:00.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:04:00.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:04:00.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:04:00.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:04:00.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:04:00.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:04:00.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:04:00.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:04:00.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:04:00.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:04:00.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.308319 32208 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:04:00.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.308923 32208 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:04:00.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.309126 32208 kserver.cc:163] Server-wide thread pool size limit: 3276
02:04:00.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.309685 32208 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:04:00.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.311084 32229 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
02:04:00.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.314214 32208 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:04:00.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.314293 32208 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.003s	user 0.000s	sys 0.000s
02:04:00.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.314354 32208 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:04:00.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.315860 32208 ts_tablet_manager.cc:616] Registered 1 tablets
02:04:00.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.315927 32208 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s	user 0.001s	sys 0.000s
02:04:00.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.316236 32229 tablet_bootstrap.cc:492] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93: Bootstrap starting.
02:04:00.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.326998 32208 rpc_server.cc:307] RPC server started. Bound to: 127.29.165.131:44535
02:04:00.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.327078 32336 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.165.131:44535 every 8 connection(s)
02:04:00.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.328032 32208 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/data/info.pb
02:04:00.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.332350 30358 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskxPorqF/build/debug/bin/kudu as pid 32208
02:04:00.332 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:04:00.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.334662 32229 log.cc:826] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93: Log is configured to *not* fsync() on all Append() calls
02:04:00.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.343104 32337 heartbeater.cc:344] Connected to a master server at 127.29.165.190:39217
02:04:00.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.343359 32337 heartbeater.cc:461] Registering TS with master...
02:04:00.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.343703 32339 heartbeater.cc:344] Connected to a master server at 127.29.165.189:39733
02:04:00.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.343784 32339 heartbeater.cc:461] Registering TS with master...
02:04:00.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.343943 32337 heartbeater.cc:507] Master 127.29.165.190:39217 requested a full tablet report, sending...
02:04:00.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.343942 32339 heartbeater.cc:507] Master 127.29.165.189:39733 requested a full tablet report, sending...
02:04:00.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.345700 30400 ts_manager.cc:194] Re-registered known tserver with Master: 26ea15144933497a98a05f83b05e8f93 (127.29.165.131:44535)
02:04:00.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.345700 30472 ts_manager.cc:194] Re-registered known tserver with Master: 26ea15144933497a98a05f83b05e8f93 (127.29.165.131:44535)
02:04:00.347 [DEBUG - kudu-nio-5] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 24 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956, tablet=830b8d1af48f4d34b281a548f1dbe5ef, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:04:00.347 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.29.165.129:45737
02:04:00.348 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

02:04:00.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.348196 30358 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskxPorqF/build/debug/bin/kudu with pid 31853
02:04:00.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.348223 32338 heartbeater.cc:344] Connected to a master server at 127.29.165.188:38583
02:04:00.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.348886 32338 heartbeater.cc:461] Registering TS with master...
02:04:00.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.349118 32338 heartbeater.cc:507] Master 127.29.165.188:38583 requested a full tablet report, sending...
02:04:00.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.350634 30542 ts_manager.cc:194] Re-registered known tserver with Master: 26ea15144933497a98a05f83b05e8f93 (127.29.165.131:44535)
02:04:00.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.351524 30542 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.165.131:44399
02:04:00.356 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:04:00.357 [DEBUG - kudu-nio-5] (Connection.java:239) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] handling channelInactive
02:04:00.357 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] cleaning up while in state READY due to: connection closed
02:04:00.358 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: connection closed
02:04:00.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.360800 32229 tablet_bootstrap.cc:492] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=21 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:04:00.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.361147 32229 tablet_bootstrap.cc:492] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93: Bootstrap complete.
02:04:00.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.362118 32229 ts_tablet_manager.cc:1403] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93: Time spent bootstrapping tablet: real 0.046s	user 0.026s	sys 0.011s
02:04:00.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.364130 32229 raft_consensus.cc:359] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:04:00.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.364444 32229 raft_consensus.cc:740] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26ea15144933497a98a05f83b05e8f93, State: Initialized, Role: FOLLOWER
02:04:00.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.364796 32229 consensus_queue.cc:260] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 7.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:04:00.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.365461 32338 heartbeater.cc:499] Master 127.29.165.188:38583 was elected leader, sending a full tablet report...
02:04:00.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:00.365556 32229 ts_tablet_manager.cc:1434] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93: Time spent starting tablet: real 0.003s	user 0.006s	sys 0.001s
02:04:00.374 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:04:00.374 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)[L],26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535),4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)]}]
02:04:00.374 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=2, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=17), Traces: [0ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [0ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [16ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@689815981(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@65b3bd38 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1fc15568 -> wakeup thread Time-limited test)) after lookup
02:04:00.375 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] connecting to peer
02:04:00.376 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:04:00.376 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] cleaning up while in state CONNECTING due to: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:04:00.376 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:04:00.393 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:04:00.394 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)[L],26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535),4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)]}]
02:04:00.394 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=4, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=37), Traces: [0ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [0ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [16ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [17ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [19ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [19ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [36ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@689815981(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@65b3bd38 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2092620094 after retry RPC@1766173140), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1fc15568 -> wakeup thread Time-limited test -> (continuation of Deferred@2092620094 after retry RPC@1766173140))) after lookup
02:04:00.394 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] connecting to peer
02:04:00.395 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:04:00.396 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] cleaning up while in state CONNECTING due to: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:04:00.396 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:04:00.435 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:04:00.435 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)[L],26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535),4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)]}]
02:04:00.435 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=6, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=78), Traces: [0ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [0ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [16ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [17ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [19ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [19ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [36ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [37ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [39ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [39ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [75ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [77ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@689815981(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@65b3bd38 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2092620094 after retry RPC@1766173140) -> (continuation of Deferred@968520634 after retry RPC@2104845904), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1fc15568 -> wakeup thread Time-limited test -> (continuation of Deferred@2092620094 after retry RPC@1766173140) -> (continuation of Deferred@968520634 after retry RPC@2104845904))) after lookup
02:04:00.436 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] connecting to peer
02:04:00.438 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:04:00.438 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] cleaning up while in state CONNECTING due to: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:04:00.438 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:04:00.514 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:04:00.515 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)[L],26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535),4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)]}]
02:04:00.515 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=8, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=158), Traces: [0ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [0ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [16ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [17ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [19ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [19ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [36ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [37ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [39ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [39ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [75ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [77ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [79ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [81ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [81ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [154ms] refreshing cache from master, [154ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [157ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@689815981(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@65b3bd38 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2092620094 after retry RPC@1766173140) -> (continuation of Deferred@968520634 after retry RPC@2104845904) -> (continuation of Deferred@845916509 after retry RPC@1694113446), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1fc15568 -> wakeup thread Time-limited test -> (continuation of Deferred@2092620094 after retry RPC@1766173140) -> (continuation of Deferred@968520634 after retry RPC@2104845904) -> (continuation of Deferred@845916509 after retry RPC@1694113446))) after lookup
02:04:00.516 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] connecting to peer
02:04:00.517 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:04:00.517 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] cleaning up while in state CONNECTING due to: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:04:00.517 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:04:00.714 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:04:00.714 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)[L],26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535),4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)]}]
02:04:00.715 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=10, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=358), Traces: [0ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [0ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [16ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [17ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [19ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [19ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [36ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [37ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [39ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [39ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [75ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [77ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [79ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [81ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [81ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [154ms] refreshing cache from master, [154ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [157ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [159ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [160ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [161ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [354ms] refreshing cache from master, [354ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [357ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@689815981(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@65b3bd38 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2092620094 after retry RPC@1766173140) -> (continuation of Deferred@968520634 after retry RPC@2104845904) -> (continuation of Deferred@845916509 after retry RPC@1694113446) -> (continuation of Deferred@255697060 after retry RPC@159005681), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1fc15568 -> wakeup thread Time-limited test -> (continuation of Deferred@2092620094 after retry RPC@1766173140) -> (continuation of Deferred@968520634 after retry RPC@2104845904) -> (continuation of Deferred@845916509 after retry RPC@1694113446) -> (continuation of Deferred@255697060 after retry RPC@159005681))) after lookup
02:04:00.715 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] connecting to peer
02:04:00.716 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:04:00.716 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] cleaning up while in state CONNECTING due to: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:04:00.716 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737
02:04:01.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:01.534459 32351 raft_consensus.cc:493] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
02:04:01.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:01.534638 32351 raft_consensus.cc:515] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:04:01.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:01.535907 32351 leader_election.cc:290] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130:35181), 06f0647651a749f8a533afd451fe5fcb (127.29.165.129:45737)
02:04:01.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:04:01.536635 32226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.165.129:45737: connect: Connection refused (error 111)
02:04:01.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:04:01.537462 32226 leader_election.cc:336] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 06f0647651a749f8a533afd451fe5fcb (127.29.165.129:45737): Network error: Client connection negotiation failed: client connection to 127.29.165.129:45737: connect: Connection refused (error 111)
02:04:01.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:01.540259 32122 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "830b8d1af48f4d34b281a548f1dbe5ef" candidate_uuid: "26ea15144933497a98a05f83b05e8f93" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" is_pre_election: true
02:04:01.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:01.540962 32225 leader_election.cc:304] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26ea15144933497a98a05f83b05e8f93; no voters: 06f0647651a749f8a533afd451fe5fcb, 4e39ec54c2d642feafa81dc5d4f066c2
02:04:01.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:01.541205 32351 raft_consensus.cc:2749] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
02:04:01.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:01.666182 32356 raft_consensus.cc:493] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 06f0647651a749f8a533afd451fe5fcb)
02:04:01.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:01.666306 32356 raft_consensus.cc:515] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:04:01.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:01.667714 32356 leader_election.cc:290] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 06f0647651a749f8a533afd451fe5fcb (127.29.165.129:45737), 26ea15144933497a98a05f83b05e8f93 (127.29.165.131:44535)
02:04:01.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:04:01.668437 32057 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.165.129:45737: connect: Connection refused (error 111)
02:04:01.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:04:01.669757 32057 leader_election.cc:336] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 06f0647651a749f8a533afd451fe5fcb (127.29.165.129:45737): Network error: Client connection negotiation failed: client connection to 127.29.165.129:45737: connect: Connection refused (error 111)
02:04:01.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:01.674180 32291 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "830b8d1af48f4d34b281a548f1dbe5ef" candidate_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "26ea15144933497a98a05f83b05e8f93" is_pre_election: true
02:04:01.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:01.674405 32291 raft_consensus.cc:2468] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e39ec54c2d642feafa81dc5d4f066c2 in term 7.
02:04:01.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:01.675154 32056 leader_election.cc:304] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26ea15144933497a98a05f83b05e8f93, 4e39ec54c2d642feafa81dc5d4f066c2; no voters: 06f0647651a749f8a533afd451fe5fcb
02:04:01.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:01.675431 32356 raft_consensus.cc:2804] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 8 FOLLOWER]: Leader pre-election won for term 9
02:04:01.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:01.675539 32356 raft_consensus.cc:493] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 06f0647651a749f8a533afd451fe5fcb)
02:04:01.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:01.675592 32356 raft_consensus.cc:3060] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 8 FOLLOWER]: Advancing to term 9
02:04:01.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:01.676455 32356 raft_consensus.cc:515] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:04:01.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:01.676918 32356 leader_election.cc:290] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [CANDIDATE]: Term 9 election: Requested vote from peers 06f0647651a749f8a533afd451fe5fcb (127.29.165.129:45737), 26ea15144933497a98a05f83b05e8f93 (127.29.165.131:44535)
02:04:01.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:01.677345 32291 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "830b8d1af48f4d34b281a548f1dbe5ef" candidate_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "26ea15144933497a98a05f83b05e8f93"
02:04:01.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:01.677503 32291 raft_consensus.cc:3060] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 7 FOLLOWER]: Advancing to term 9
02:04:01.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:04:01.678102 32057 leader_election.cc:336] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 06f0647651a749f8a533afd451fe5fcb (127.29.165.129:45737): Network error: Client connection negotiation failed: client connection to 127.29.165.129:45737: connect: Connection refused (error 111)
02:04:01.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:01.679374 32291 raft_consensus.cc:2468] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e39ec54c2d642feafa81dc5d4f066c2 in term 9.
02:04:01.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:01.679723 32056 leader_election.cc:304] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26ea15144933497a98a05f83b05e8f93, 4e39ec54c2d642feafa81dc5d4f066c2; no voters: 06f0647651a749f8a533afd451fe5fcb
02:04:01.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:01.679893 32356 raft_consensus.cc:2804] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 9 FOLLOWER]: Leader election won for term 9
02:04:01.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:01.680130 32356 raft_consensus.cc:697] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 9 LEADER]: Becoming Leader. State: Replica: 4e39ec54c2d642feafa81dc5d4f066c2, State: Running, Role: LEADER
02:04:01.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:01.680428 32356 consensus_queue.cc:237] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 8.32, Last appended by leader: 32, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:04:01.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:01.683276 30542 catalog_manager.cc:5696] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 reported cstate change: term changed from 8 to 9, leader changed from 06f0647651a749f8a533afd451fe5fcb (127.29.165.129) to 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130). New cstate: current_term: 9 leader_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } health_report { overall_health: UNKNOWN } } }
02:04:01.934 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:04:01.935 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737),26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535),4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)[L]]}]
02:04:01.935 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=12, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=1578), Traces: [0ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [0ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [14ms] refreshing cache from master, [14ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [16ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [17ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [19ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [19ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [36ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [37ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [39ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [39ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [75ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [77ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [79ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [81ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [81ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [154ms] refreshing cache from master, [154ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [157ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [159ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [160ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [161ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [354ms] refreshing cache from master, [354ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [357ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [358ms] sending RPC to server 06f0647651a749f8a533afd451fe5fcb, [359ms] received response from server 06f0647651a749f8a533afd451fe5fcb: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [359ms] delaying RPC due to: Network error: Failed to connect to peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737): Connection refused: /127.29.165.129:45737, [1574ms] refreshing cache from master, [1575ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [1577ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@689815981(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@65b3bd38 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2092620094 after retry RPC@1766173140) -> (continuation of Deferred@968520634 after retry RPC@2104845904) -> (continuation of Deferred@845916509 after retry RPC@1694113446) -> (continuation of Deferred@255697060 after retry RPC@159005681) -> (continuation of Deferred@113505362 after retry RPC@1581183688), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1fc15568 -> wakeup thread Time-limited test -> (continuation of Deferred@2092620094 after retry RPC@1766173140) -> (continuation of Deferred@968520634 after retry RPC@2104845904) -> (continuation of Deferred@845916509 after retry RPC@1694113446) -> (continuation of Deferred@255697060 after retry RPC@159005681) -> (continuation of Deferred@113505362 after retry RPC@1581183688))) after lookup
02:04:01.936 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] connecting to peer
02:04:01.936 [DEBUG - kudu-nio-3] (Connection.java:757) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] Successfully connected to peer
02:04:01.949 [DEBUG - kudu-nio-3] (Negotiator.java:919) Authenticated connection [id: 0x2f55901b, L:/127.0.0.1:41382 - R:/127.29.165.130:35181] using TOKEN/null
02:04:01.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:01.953097 32291 raft_consensus.cc:1275] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 9 FOLLOWER]: Refusing update from remote peer 4e39ec54c2d642feafa81dc5d4f066c2: Log matching property violated. Preceding OpId in replica: term: 7 index: 28. Preceding OpId from leader: term: 9 index: 34. (index mismatch)
02:04:01.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:04:01.953320 32057 consensus_peers.cc:597] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 -> Peer 06f0647651a749f8a533afd451fe5fcb (127.29.165.129:45737): Couldn't send request to peer 06f0647651a749f8a533afd451fe5fcb. Status: Network error: Client connection negotiation failed: client connection to 127.29.165.129:45737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:04:01.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:01.953749 32356 consensus_queue.cc:1048] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 28, Time since last communication: 0.000s
02:04:01.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:01.963336 32375 mvcc.cc:204] Tried to move back new op lower bound from 7287070072633565184 to 7287070071525933056. Current Snapshot: MvccSnapshot[applied={T|T < 7287070072633565184}]
02:04:01.976 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.29.165.129:45737
02:04:01.977 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

02:04:01.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:01.977566 30358 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskxPorqF/build/debug/bin/kudu
02:04:01.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskxPorqF/build/debug/bin/kudu
02:04:01.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/wal
02:04:01.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/data
02:04:01.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:04:01.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:04:01.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:04:01.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:04:01.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:04:01.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:04:01.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/logs
02:04:01.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/data/info.pb
02:04:01.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:04:01.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:04:01.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:04:01.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:04:01.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:04:01.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:04:01.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:04:01.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:04:01.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:04:01.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:04:01.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.165.129:45737
02:04:01.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.165.129
02:04:01.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.165.190:39217,127.29.165.189:39733,127.29.165.188:38583
02:04:01.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=46663
02:04:01.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.165.129
02:04:01.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.165.148:38003
02:04:01.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:04:01.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:04:01.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:04:02.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:04:02.078858 32382 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:04:02.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:04:02.079095 32382 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:04:02.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:04:02.079157 32382 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:04:02.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:04:02.082435 32382 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:04:02.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:04:02.082584 32382 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.165.129
02:04:02.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.086429 32382 tablet_server_runner.cc:78] Tablet server non-default flags:
02:04:02.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:04:02.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.165.148:38003
02:04:02.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:04:02.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:04:02.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/data
02:04:02.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/wal
02:04:02.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:04:02.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:04:02.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.165.129:45737
02:04:02.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:04:02.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:04:02.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:04:02.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/data/info.pb
02:04:02.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.165.129
02:04:02.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=46663
02:04:02.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.165.190:39217,127.29.165.189:39733,127.29.165.188:38583
02:04:02.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:04:02.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.32382
02:04:02.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:04:02.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:04:02.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:04:02.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:04:02.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.165.129
02:04:02.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/logs
02:04:02.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:04:02.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:04:02.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:04:02.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:04:02.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:04:02.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
02:04:02.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type DEBUG
02:04:02.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 01:43:15 UTC on bdcb31816ec0
02:04:02.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11948
02:04:02.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.087848 32382 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:04:02.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.089020 32382 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:04:02.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:04:02.095019 32387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:04:02.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:04:02.095039 32388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:04:02.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:04:02.095118 32390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:04:02.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.095295 32382 server_base.cc:1061] running on GCE node
02:04:02.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.095933 32382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:04:02.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.096455 32382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:04:02.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.097653 32382 hybrid_clock.cc:648] HybridClock initialized: now 1779069842097636 us; error 31 us; skew 500 ppm
02:04:02.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.099654 32382 webserver.cc:533] Webserver started at http://127.29.165.129:46663/ using document root <none> and password file <none>
02:04:02.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.100239 32382 fs_manager.cc:362] Metadata directory not provided
02:04:02.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.100329 32382 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:04:02.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.103668 32382 fs_manager.cc:714] Time spent opening directory manager: real 0.002s	user 0.002s	sys 0.001s
02:04:02.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.105485 32396 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:04:02.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.106478 32382 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
02:04:02.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.106600 32382 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/data,/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/wal
02:04:02.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "06f0647651a749f8a533afd451fe5fcb"
02:04:02.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:03:36 on dist-test-slave-838s"
02:04:02.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.107074 32382 fs_report.cc:389] FS layout report
02:04:02.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:04:02.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/wal
02:04:02.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/wal
02:04:02.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/data/data
02:04:02.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:04:02.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:04:02.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:04:02.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:04:02.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:04:02.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:04:02.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:04:02.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:04:02.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:04:02.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:04:02.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:04:02.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.128772 32382 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:04:02.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.129421 32382 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:04:02.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.129629 32382 kserver.cc:163] Server-wide thread pool size limit: 3276
02:04:02.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.130214 32382 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:04:02.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.131624 32403 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
02:04:02.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.135239 32382 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:04:02.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.135309 32382 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.004s	user 0.001s	sys 0.000s
02:04:02.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.135371 32382 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:04:02.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.136788 32382 ts_tablet_manager.cc:616] Registered 1 tablets
02:04:02.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.136855 32382 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s	user 0.000s	sys 0.000s
02:04:02.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.137152 32403 tablet_bootstrap.cc:492] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb: Bootstrap starting.
02:04:02.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.148196 32382 rpc_server.cc:307] RPC server started. Bound to: 127.29.165.129:45737
02:04:02.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.148211 32510 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.165.129:45737 every 8 connection(s)
02:04:02.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.149223 32382 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-0/data/info.pb
02:04:02.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.153754 30358 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskxPorqF/build/debug/bin/kudu as pid 32382
02:04:02.154 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:04:02.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.155318 32403 log.cc:826] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb: Log is configured to *not* fsync() on all Append() calls
02:04:02.163 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 27 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956, tablet=830b8d1af48f4d34b281a548f1dbe5ef, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:04:02.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.161629 32511 heartbeater.cc:344] Connected to a master server at 127.29.165.190:39217
02:04:02.163 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.29.165.130:35181
02:04:02.164 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

02:04:02.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.161855 32511 heartbeater.cc:461] Registering TS with master...
02:04:02.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.162384 32511 heartbeater.cc:507] Master 127.29.165.190:39217 requested a full tablet report, sending...
02:04:02.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.163632 32513 heartbeater.cc:344] Connected to a master server at 127.29.165.189:39733
02:04:02.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.163730 32513 heartbeater.cc:461] Registering TS with master...
02:04:02.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.163954 32513 heartbeater.cc:507] Master 127.29.165.189:39733 requested a full tablet report, sending...
02:04:02.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.164240 30400 ts_manager.cc:194] Re-registered known tserver with Master: 06f0647651a749f8a533afd451fe5fcb (127.29.165.129:45737)
02:04:02.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.164530 30358 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskxPorqF/build/debug/bin/kudu with pid 32039
02:04:02.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.165513 30472 ts_manager.cc:194] Re-registered known tserver with Master: 06f0647651a749f8a533afd451fe5fcb (127.29.165.129:45737)
02:04:02.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.161628 32512 heartbeater.cc:344] Connected to a master server at 127.29.165.188:38583
02:04:02.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.174124 32512 heartbeater.cc:461] Registering TS with master...
02:04:02.174 [DEBUG - kudu-nio-3] (Connection.java:239) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] handling channelInactive
02:04:02.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.174345 32512 heartbeater.cc:507] Master 127.29.165.188:38583 requested a full tablet report, sending...
02:04:02.174 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] cleaning up while in state READY due to: connection closed
02:04:02.175 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:04:02.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.175438 30542 ts_manager.cc:194] Re-registered known tserver with Master: 06f0647651a749f8a533afd451fe5fcb (127.29.165.129:45737)
02:04:02.176 [DEBUG - Time-limited test] (Connection.java:748) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] connecting to peer
02:04:02.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.176360 30542 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.165.129:52063
02:04:02.177 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:04:02.177 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] cleaning up while in state CONNECTING due to: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:04:02.177 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:04:02.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.184547 32403 tablet_bootstrap.cc:492] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=24 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:04:02.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.184996 32403 tablet_bootstrap.cc:492] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb: Bootstrap complete.
02:04:02.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.186049 32403 ts_tablet_manager.cc:1403] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb: Time spent bootstrapping tablet: real 0.049s	user 0.032s	sys 0.009s
02:04:02.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.188472 32403 raft_consensus.cc:359] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:04:02.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.188918 32403 raft_consensus.cc:740] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06f0647651a749f8a533afd451fe5fcb, State: Initialized, Role: FOLLOWER
02:04:02.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.189435 32403 consensus_queue.cc:260] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 8.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:04:02.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.190186 32512 heartbeater.cc:499] Master 127.29.165.188:38583 was elected leader, sending a full tablet report...
02:04:02.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:02.190452 32403 ts_tablet_manager.cc:1434] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb: Time spent starting tablet: real 0.004s	user 0.007s	sys 0.000s
02:04:02.194 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:04:02.194 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737),26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535),4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)[L]]}]
02:04:02.194 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=2, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=19), Traces: [0ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [2ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [2ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [16ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [19ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@826562473(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@18c55bfb -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7fa05b36 -> wakeup thread Time-limited test)) after lookup
02:04:02.195 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] connecting to peer
02:04:02.196 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:04:02.196 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] cleaning up while in state CONNECTING due to: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:04:02.196 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:04:02.214 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:04:02.214 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737),26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535),4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)[L]]}]
02:04:02.215 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=4, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=39), Traces: [0ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [2ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [2ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [16ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [19ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [20ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [21ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [21ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [39ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@826562473(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@18c55bfb -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@346570494 after retry RPC@1764535990), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7fa05b36 -> wakeup thread Time-limited test -> (continuation of Deferred@346570494 after retry RPC@1764535990))) after lookup
02:04:02.215 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] connecting to peer
02:04:02.216 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:04:02.217 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] cleaning up while in state CONNECTING due to: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:04:02.217 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:04:02.233 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:04:02.233 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737),26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535),4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)[L]]}]
02:04:02.234 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=6, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=58), Traces: [0ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [2ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [2ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [16ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [19ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [20ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [21ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [21ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [39ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [40ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [42ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [42ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [56ms] refreshing cache from master, [56ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [58ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@826562473(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@18c55bfb -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@346570494 after retry RPC@1764535990) -> (continuation of Deferred@2022204647 after retry RPC@1952263262), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7fa05b36 -> wakeup thread Time-limited test -> (continuation of Deferred@346570494 after retry RPC@1764535990) -> (continuation of Deferred@2022204647 after retry RPC@1952263262))) after lookup
02:04:02.234 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] connecting to peer
02:04:02.236 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:04:02.236 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] cleaning up while in state CONNECTING due to: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:04:02.236 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:04:02.293 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:04:02.294 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737),26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535),4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)[L]]}]
02:04:02.294 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=8, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=118), Traces: [0ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [2ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [2ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [16ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [19ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [20ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [21ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [21ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [39ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [40ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [42ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [42ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [56ms] refreshing cache from master, [56ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [58ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [59ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [61ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [61ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [116ms] refreshing cache from master, [116ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [118ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@826562473(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@18c55bfb -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@346570494 after retry RPC@1764535990) -> (continuation of Deferred@2022204647 after retry RPC@1952263262) -> (continuation of Deferred@618342307 after retry RPC@972673210), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7fa05b36 -> wakeup thread Time-limited test -> (continuation of Deferred@346570494 after retry RPC@1764535990) -> (continuation of Deferred@2022204647 after retry RPC@1952263262) -> (continuation of Deferred@618342307 after retry RPC@972673210))) after lookup
02:04:02.295 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] connecting to peer
02:04:02.295 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:04:02.295 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] cleaning up while in state CONNECTING due to: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:04:02.296 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:04:02.314 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:04:02.314 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737),26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535),4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)[L]]}]
02:04:02.314 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=10, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=138), Traces: [0ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [2ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [2ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [16ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [19ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [20ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [21ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [21ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [39ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [40ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [42ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [42ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [56ms] refreshing cache from master, [56ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [58ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [59ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [61ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [61ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [116ms] refreshing cache from master, [116ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [118ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [119ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [121ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [121ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [136ms] refreshing cache from master, [136ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [138ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@826562473(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@18c55bfb -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@346570494 after retry RPC@1764535990) -> (continuation of Deferred@2022204647 after retry RPC@1952263262) -> (continuation of Deferred@618342307 after retry RPC@972673210) -> (continuation of Deferred@542442751 after retry RPC@1465809999), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7fa05b36 -> wakeup thread Time-limited test -> (continuation of Deferred@346570494 after retry RPC@1764535990) -> (continuation of Deferred@2022204647 after retry RPC@1952263262) -> (continuation of Deferred@618342307 after retry RPC@972673210) -> (continuation of Deferred@542442751 after retry RPC@1465809999))) after lookup
02:04:02.315 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] connecting to peer
02:04:02.315 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:04:02.316 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] cleaning up while in state CONNECTING due to: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:04:02.316 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:04:03.334 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:04:03.334 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737),26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535),4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)[L]]}]
02:04:03.335 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=12, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=1159), Traces: [0ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [2ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [2ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [16ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [19ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [20ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [21ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [21ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [39ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [40ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [42ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [42ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [56ms] refreshing cache from master, [56ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [58ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [59ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [61ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [61ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [116ms] refreshing cache from master, [116ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [118ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [119ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [121ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [121ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [136ms] refreshing cache from master, [136ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [138ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [139ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [141ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [141ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [1156ms] refreshing cache from master, [1156ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [1159ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@826562473(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@18c55bfb -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@346570494 after retry RPC@1764535990) -> (continuation of Deferred@2022204647 after retry RPC@1952263262) -> (continuation of Deferred@618342307 after retry RPC@972673210) -> (continuation of Deferred@542442751 after retry RPC@1465809999) -> (continuation of Deferred@768164429 after retry RPC@587565423), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7fa05b36 -> wakeup thread Time-limited test -> (continuation of Deferred@346570494 after retry RPC@1764535990) -> (continuation of Deferred@2022204647 after retry RPC@1952263262) -> (continuation of Deferred@618342307 after retry RPC@972673210) -> (continuation of Deferred@542442751 after retry RPC@1465809999) -> (continuation of Deferred@768164429 after retry RPC@587565423))) after lookup
02:04:03.336 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] connecting to peer
02:04:03.337 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:04:03.337 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] cleaning up while in state CONNECTING due to: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:04:03.337 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181
02:04:03.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:03.457501 32526 raft_consensus.cc:493] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
02:04:03.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:03.457677 32526 raft_consensus.cc:515] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:04:03.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:03.458830 32526 leader_election.cc:290] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130:35181), 26ea15144933497a98a05f83b05e8f93 (127.29.165.131:44535)
02:04:03.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:04:03.459462 32399 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.165.130:35181: connect: Connection refused (error 111)
02:04:03.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:04:03.460417 32399 leader_election.cc:336] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130:35181): Network error: Client connection negotiation failed: client connection to 127.29.165.130:35181: connect: Connection refused (error 111)
02:04:03.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:03.463163 32290 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "830b8d1af48f4d34b281a548f1dbe5ef" candidate_uuid: "06f0647651a749f8a533afd451fe5fcb" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "26ea15144933497a98a05f83b05e8f93" is_pre_election: true
02:04:03.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:03.463660 32399 leader_election.cc:304] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 06f0647651a749f8a533afd451fe5fcb; no voters: 26ea15144933497a98a05f83b05e8f93, 4e39ec54c2d642feafa81dc5d4f066c2
02:04:03.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:03.463876 32526 raft_consensus.cc:2749] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
02:04:03.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:03.476843 32531 raft_consensus.cc:493] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4e39ec54c2d642feafa81dc5d4f066c2)
02:04:03.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:03.476976 32531 raft_consensus.cc:515] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:04:03.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:03.477437 32531 leader_election.cc:290] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130:35181), 06f0647651a749f8a533afd451fe5fcb (127.29.165.129:45737)
02:04:03.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:04:03.478670 32225 leader_election.cc:336] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130:35181): Network error: Client connection negotiation failed: client connection to 127.29.165.130:35181: connect: Connection refused (error 111)
02:04:03.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:03.481657 32465 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "830b8d1af48f4d34b281a548f1dbe5ef" candidate_uuid: "26ea15144933497a98a05f83b05e8f93" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "06f0647651a749f8a533afd451fe5fcb" is_pre_election: true
02:04:03.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:03.481860 32465 raft_consensus.cc:2468] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26ea15144933497a98a05f83b05e8f93 in term 8.
02:04:03.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:03.482756 32226 leader_election.cc:304] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06f0647651a749f8a533afd451fe5fcb, 26ea15144933497a98a05f83b05e8f93; no voters: 4e39ec54c2d642feafa81dc5d4f066c2
02:04:03.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:03.482993 32531 raft_consensus.cc:2804] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 9 FOLLOWER]: Leader pre-election won for term 10
02:04:03.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:03.483067 32531 raft_consensus.cc:493] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 4e39ec54c2d642feafa81dc5d4f066c2)
02:04:03.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:03.483119 32531 raft_consensus.cc:3060] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 9 FOLLOWER]: Advancing to term 10
02:04:03.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:03.484179 32531 raft_consensus.cc:515] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:04:03.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:03.484652 32531 leader_election.cc:290] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [CANDIDATE]: Term 10 election: Requested vote from peers 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130:35181), 06f0647651a749f8a533afd451fe5fcb (127.29.165.129:45737)
02:04:03.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:03.485096 32465 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "830b8d1af48f4d34b281a548f1dbe5ef" candidate_uuid: "26ea15144933497a98a05f83b05e8f93" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "06f0647651a749f8a533afd451fe5fcb"
02:04:03.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:03.485239 32465 raft_consensus.cc:3060] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 8 FOLLOWER]: Advancing to term 10
02:04:03.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:03.487372 32465 raft_consensus.cc:2468] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26ea15144933497a98a05f83b05e8f93 in term 10.
02:04:03.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:04:03.487425 32225 leader_election.cc:336] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130:35181): Network error: Client connection negotiation failed: client connection to 127.29.165.130:35181: connect: Connection refused (error 111)
02:04:03.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:03.487680 32226 leader_election.cc:304] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06f0647651a749f8a533afd451fe5fcb, 26ea15144933497a98a05f83b05e8f93; no voters: 4e39ec54c2d642feafa81dc5d4f066c2
02:04:03.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:03.487905 32531 raft_consensus.cc:2804] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 10 FOLLOWER]: Leader election won for term 10
02:04:03.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:03.488137 32531 raft_consensus.cc:697] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [term 10 LEADER]: Becoming Leader. State: Replica: 26ea15144933497a98a05f83b05e8f93, State: Running, Role: LEADER
02:04:03.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:03.488436 32531 consensus_queue.cc:237] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 9.36, Last appended by leader: 36, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:04:03.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:03.491189 30542 catalog_manager.cc:5696] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 reported cstate change: term changed from 9 to 10, leader changed from 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130) to 26ea15144933497a98a05f83b05e8f93 (127.29.165.131). New cstate: current_term: 10 leader_uuid: "26ea15144933497a98a05f83b05e8f93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } health_report { overall_health: HEALTHY } } }
02:04:03.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:04:03.955724 32225 consensus_peers.cc:597] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 -> Peer 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130:35181): Couldn't send request to peer 4e39ec54c2d642feafa81dc5d4f066c2. Status: Network error: Client connection negotiation failed: client connection to 127.29.165.130:35181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:04:03.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:03.997637 32465 raft_consensus.cc:1275] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 10 FOLLOWER]: Refusing update from remote peer 26ea15144933497a98a05f83b05e8f93: Log matching property violated. Preceding OpId in replica: term: 8 index: 32. Preceding OpId from leader: term: 10 index: 37. (index mismatch)
02:04:03.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:03.998395 32531 consensus_queue.cc:1048] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 32, Time since last communication: 0.000s
02:04:05.634 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:04:05.634 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737),26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)[L],4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)]}]
02:04:05.634 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=14, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=3458), Traces: [0ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [2ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [2ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [16ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [19ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [20ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [21ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [21ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [36ms] refreshing cache from master, [36ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [39ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [40ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [42ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [42ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [56ms] refreshing cache from master, [56ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [58ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [59ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [61ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [61ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [116ms] refreshing cache from master, [116ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [118ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [119ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [121ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [121ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [136ms] refreshing cache from master, [136ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [138ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [139ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [141ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [141ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [1156ms] refreshing cache from master, [1156ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [1159ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [1160ms] sending RPC to server 4e39ec54c2d642feafa81dc5d4f066c2, [1162ms] received response from server 4e39ec54c2d642feafa81dc5d4f066c2: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [1162ms] delaying RPC due to: Network error: Failed to connect to peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181): Connection refused: /127.29.165.130:35181, [3456ms] refreshing cache from master, [3456ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [3458ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@826562473(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@18c55bfb -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@346570494 after retry RPC@1764535990) -> (continuation of Deferred@2022204647 after retry RPC@1952263262) -> (continuation of Deferred@618342307 after retry RPC@972673210) -> (continuation of Deferred@542442751 after retry RPC@1465809999) -> (continuation of Deferred@768164429 after retry RPC@587565423) -> (continuation of Deferred@624290123 after retry RPC@720654963), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7fa05b36 -> wakeup thread Time-limited test -> (continuation of Deferred@346570494 after retry RPC@1764535990) -> (continuation of Deferred@2022204647 after retry RPC@1952263262) -> (continuation of Deferred@618342307 after retry RPC@972673210) -> (continuation of Deferred@542442751 after retry RPC@1465809999) -> (continuation of Deferred@768164429 after retry RPC@587565423) -> (continuation of Deferred@624290123 after retry RPC@720654963))) after lookup
02:04:05.635 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] connecting to peer
02:04:05.636 [DEBUG - kudu-nio-3] (Connection.java:757) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] Successfully connected to peer
02:04:05.653 [DEBUG - kudu-nio-3] (Negotiator.java:919) Authenticated connection [id: 0x49cb42a1, L:/127.0.0.1:59228 - R:/127.29.165.131:44535] using TOKEN/null
02:04:05.672 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.29.165.130:35181
02:04:05.673 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

02:04:05.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.673804 30358 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskxPorqF/build/debug/bin/kudu
02:04:05.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskxPorqF/build/debug/bin/kudu
02:04:05.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/wal
02:04:05.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/data
02:04:05.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:04:05.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:04:05.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:04:05.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:04:05.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:04:05.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:04:05.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/logs
02:04:05.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/data/info.pb
02:04:05.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:04:05.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:04:05.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:04:05.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:04:05.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:04:05.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:04:05.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:04:05.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:04:05.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:04:05.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:04:05.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.165.130:35181
02:04:05.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.165.130
02:04:05.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.165.190:39217,127.29.165.189:39733,127.29.165.188:38583
02:04:05.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=33955
02:04:05.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.165.130
02:04:05.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.165.148:38003
02:04:05.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:04:05.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:04:05.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:04:05.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:04:05.774770 32564 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:04:05.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:04:05.775038 32564 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:04:05.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:04:05.775102 32564 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:04:05.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:04:05.778621 32564 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:04:05.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:04:05.778806 32564 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.165.130
02:04:05.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.782598 32564 tablet_server_runner.cc:78] Tablet server non-default flags:
02:04:05.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:04:05.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.165.148:38003
02:04:05.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:04:05.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:04:05.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/data
02:04:05.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/wal
02:04:05.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:04:05.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:04:05.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.165.130:35181
02:04:05.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:04:05.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:04:05.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:04:05.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/data/info.pb
02:04:05.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.165.130
02:04:05.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=33955
02:04:05.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.165.190:39217,127.29.165.189:39733,127.29.165.188:38583
02:04:05.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:04:05.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.32564
02:04:05.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:04:05.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:04:05.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:04:05.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:04:05.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.165.130
02:04:05.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/logs
02:04:05.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:04:05.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:04:05.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:04:05.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:04:05.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:04:05.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
02:04:05.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type DEBUG
02:04:05.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 01:43:15 UTC on bdcb31816ec0
02:04:05.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11948
02:04:05.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.784029 32564 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:04:05.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.785259 32564 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:04:05.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:04:05.791395 32569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:04:05.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.791720 32564 server_base.cc:1061] running on GCE node
02:04:05.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:04:05.791440 32572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:04:05.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:04:05.791451 32570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:04:05.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.792367 32564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:04:05.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.792972 32564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:04:05.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.794153 32564 hybrid_clock.cc:648] HybridClock initialized: now 1779069845794142 us; error 33 us; skew 500 ppm
02:04:05.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.795922 32564 webserver.cc:533] Webserver started at http://127.29.165.130:33955/ using document root <none> and password file <none>
02:04:05.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.796509 32564 fs_manager.cc:362] Metadata directory not provided
02:04:05.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.796595 32564 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:04:05.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.799753 32564 fs_manager.cc:714] Time spent opening directory manager: real 0.002s	user 0.001s	sys 0.002s
02:04:05.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.801600 32578 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:04:05.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.802557 32564 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
02:04:05.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.802677 32564 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/data,/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/wal
02:04:05.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "4e39ec54c2d642feafa81dc5d4f066c2"
02:04:05.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:03:36 on dist-test-slave-838s"
02:04:05.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.803207 32564 fs_report.cc:389] FS layout report
02:04:05.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:04:05.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/wal
02:04:05.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/wal
02:04:05.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/data/data
02:04:05.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:04:05.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:04:05.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:04:05.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:04:05.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:04:05.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:04:05.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:04:05.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:04:05.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:04:05.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:04:05.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:04:05.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.829120 32564 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:04:05.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.829872 32564 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:04:05.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.830091 32564 kserver.cc:163] Server-wide thread pool size limit: 3276
02:04:05.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.830684 32564 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:04:05.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.832027 32585 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
02:04:05.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.835314 32564 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:04:05.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.835386 32564 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.004s	user 0.000s	sys 0.000s
02:04:05.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.835448 32564 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:04:05.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.836930 32564 ts_tablet_manager.cc:616] Registered 1 tablets
02:04:05.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.836995 32564 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s	user 0.000s	sys 0.000s
02:04:05.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.837306 32585 tablet_bootstrap.cc:492] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2: Bootstrap starting.
02:04:05.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.848189 32564 rpc_server.cc:307] RPC server started. Bound to: 127.29.165.130:35181
02:04:05.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.848208 32692 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.165.130:35181 every 8 connection(s)
02:04:05.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.849167 32564 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-1/data/info.pb
02:04:05.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.849711 30358 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskxPorqF/build/debug/bin/kudu as pid 32564
02:04:05.850 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:04:05.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.857740 32585 log.cc:826] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2: Log is configured to *not* fsync() on all Append() calls
02:04:05.860 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 30 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956, tablet=830b8d1af48f4d34b281a548f1dbe5ef, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:04:05.861 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.29.165.131:44535
02:04:05.862 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

02:04:05.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.862521 30358 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskxPorqF/build/debug/bin/kudu with pid 32208
02:04:05.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.868948 32694 heartbeater.cc:344] Connected to a master server at 127.29.165.188:38583
02:04:05.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.869045 32695 heartbeater.cc:344] Connected to a master server at 127.29.165.189:39733
02:04:05.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.869246 32694 heartbeater.cc:461] Registering TS with master...
02:04:05.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.869246 32695 heartbeater.cc:461] Registering TS with master...
02:04:05.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.869740 32693 heartbeater.cc:344] Connected to a master server at 127.29.165.190:39217
02:04:05.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.869822 32693 heartbeater.cc:461] Registering TS with master...
02:04:05.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.869915 32694 heartbeater.cc:507] Master 127.29.165.188:38583 requested a full tablet report, sending...
02:04:05.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.869973 32693 heartbeater.cc:507] Master 127.29.165.190:39217 requested a full tablet report, sending...
02:04:05.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.869912 32695 heartbeater.cc:507] Master 127.29.165.189:39733 requested a full tablet report, sending...
02:04:05.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.871778 30542 ts_manager.cc:194] Re-registered known tserver with Master: 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130:35181)
02:04:05.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.871861 30400 ts_manager.cc:194] Re-registered known tserver with Master: 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130:35181)
02:04:05.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.872237 30472 ts_manager.cc:194] Re-registered known tserver with Master: 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130:35181)
02:04:05.872 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:04:05.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.872782 30542 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.165.130:57965
02:04:05.873 [DEBUG - kudu-nio-3] (Connection.java:239) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] handling channelInactive
02:04:05.873 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] cleaning up while in state READY due to: connection closed
02:04:05.873 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: connection closed
02:04:05.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.884939 32585 tablet_bootstrap.cc:492] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=36 overwritten=0 applied=36 ignored=0} inserts{seen=27 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:04:05.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.885401 32585 tablet_bootstrap.cc:492] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2: Bootstrap complete.
02:04:05.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.886890 32585 ts_tablet_manager.cc:1403] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2: Time spent bootstrapping tablet: real 0.050s	user 0.030s	sys 0.012s
02:04:05.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.888919 32585 raft_consensus.cc:359] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 9 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:04:05.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.889266 32585 raft_consensus.cc:740] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e39ec54c2d642feafa81dc5d4f066c2, State: Initialized, Role: FOLLOWER
02:04:05.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.889729 32585 consensus_queue.cc:260] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 9.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:04:05.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.890396 32694 heartbeater.cc:499] Master 127.29.165.188:38583 was elected leader, sending a full tablet report...
02:04:05.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:05.890604 32585 ts_tablet_manager.cc:1434] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2: Time spent starting tablet: real 0.004s	user 0.003s	sys 0.000s
02:04:05.893 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:04:05.894 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737),26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)[L],4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)]}]
02:04:05.894 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=2, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=21), Traces: [0ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [0ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: connection closed, [0ms] delaying RPC due to: Network error: connection closed, [18ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [20ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@1600959951(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@5703139b -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@40ce4b13 -> wakeup thread Time-limited test)) after lookup
02:04:05.894 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] connecting to peer
02:04:05.895 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:04:05.895 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] cleaning up while in state CONNECTING due to: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:04:05.895 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:04:05.913 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:04:05.914 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737),26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)[L],4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)]}]
02:04:05.914 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=4, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=41), Traces: [0ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [0ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: connection closed, [0ms] delaying RPC due to: Network error: connection closed, [18ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [20ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [21ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [22ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [22ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [40ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@1600959951(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@5703139b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@728615888 after retry RPC@424254333), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@40ce4b13 -> wakeup thread Time-limited test -> (continuation of Deferred@728615888 after retry RPC@424254333))) after lookup
02:04:05.914 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] connecting to peer
02:04:05.915 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:04:05.916 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] cleaning up while in state CONNECTING due to: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:04:05.916 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:04:05.954 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:04:05.954 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737),26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)[L],4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)]}]
02:04:05.954 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=6, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=81), Traces: [0ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [0ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: connection closed, [0ms] delaying RPC due to: Network error: connection closed, [18ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [20ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [21ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [22ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [22ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [40ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [41ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [43ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [43ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [78ms] refreshing cache from master, [78ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [80ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@1600959951(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@5703139b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@728615888 after retry RPC@424254333) -> (continuation of Deferred@639368020 after retry RPC@1943805593), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@40ce4b13 -> wakeup thread Time-limited test -> (continuation of Deferred@728615888 after retry RPC@424254333) -> (continuation of Deferred@639368020 after retry RPC@1943805593))) after lookup
02:04:05.955 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] connecting to peer
02:04:05.956 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:04:05.956 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] cleaning up while in state CONNECTING due to: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:04:05.956 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:04:06.034 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:04:06.034 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737),26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)[L],4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)]}]
02:04:06.034 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=8, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=161), Traces: [0ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [0ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: connection closed, [0ms] delaying RPC due to: Network error: connection closed, [18ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [20ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [21ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [22ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [22ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [40ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [41ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [43ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [43ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [78ms] refreshing cache from master, [78ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [80ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [81ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [83ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [83ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [158ms] refreshing cache from master, [158ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [160ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@1600959951(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@5703139b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@728615888 after retry RPC@424254333) -> (continuation of Deferred@639368020 after retry RPC@1943805593) -> (continuation of Deferred@1849135097 after retry RPC@2106354800), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@40ce4b13 -> wakeup thread Time-limited test -> (continuation of Deferred@728615888 after retry RPC@424254333) -> (continuation of Deferred@639368020 after retry RPC@1943805593) -> (continuation of Deferred@1849135097 after retry RPC@2106354800))) after lookup
02:04:06.036 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] connecting to peer
02:04:06.037 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:04:06.037 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] cleaning up while in state CONNECTING due to: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:04:06.037 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:04:06.093 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:04:06.093 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737),26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)[L],4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)]}]
02:04:06.094 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=10, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=220), Traces: [0ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [0ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: connection closed, [0ms] delaying RPC due to: Network error: connection closed, [18ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [20ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [21ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [22ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [22ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [40ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [41ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [43ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [43ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [78ms] refreshing cache from master, [78ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [80ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [81ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [83ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [83ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [158ms] refreshing cache from master, [158ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [160ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [163ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [164ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [164ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [218ms] refreshing cache from master, [218ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [220ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@1600959951(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@5703139b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@728615888 after retry RPC@424254333) -> (continuation of Deferred@639368020 after retry RPC@1943805593) -> (continuation of Deferred@1849135097 after retry RPC@2106354800) -> (continuation of Deferred@1478225759 after retry RPC@1189799872), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@40ce4b13 -> wakeup thread Time-limited test -> (continuation of Deferred@728615888 after retry RPC@424254333) -> (continuation of Deferred@639368020 after retry RPC@1943805593) -> (continuation of Deferred@1849135097 after retry RPC@2106354800) -> (continuation of Deferred@1478225759 after retry RPC@1189799872))) after lookup
02:04:06.094 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] connecting to peer
02:04:06.095 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:04:06.095 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] cleaning up while in state CONNECTING due to: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:04:06.096 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535) for tablet 830b8d1af48f4d34b281a548f1dbe5ef: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535
02:04:07.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.172729 32706 raft_consensus.cc:493] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 26ea15144933497a98a05f83b05e8f93)
02:04:07.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.172861 32706 raft_consensus.cc:515] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:04:07.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.173286 32706 leader_election.cc:290] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130:35181), 26ea15144933497a98a05f83b05e8f93 (127.29.165.131:44535)
02:04:07.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:04:07.175771 32399 leader_election.cc:336] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 26ea15144933497a98a05f83b05e8f93 (127.29.165.131:44535): Network error: Client connection negotiation failed: client connection to 127.29.165.131:44535: connect: Connection refused (error 111)
02:04:07.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.178877 32647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "830b8d1af48f4d34b281a548f1dbe5ef" candidate_uuid: "06f0647651a749f8a533afd451fe5fcb" candidate_term: 11 candidate_status { last_received { term: 10 index: 40 } } ignore_live_leader: false dest_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" is_pre_election: true
02:04:07.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.179133 32647 raft_consensus.cc:2468] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06f0647651a749f8a533afd451fe5fcb in term 9.
02:04:07.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.179811 32399 leader_election.cc:304] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06f0647651a749f8a533afd451fe5fcb, 4e39ec54c2d642feafa81dc5d4f066c2; no voters: 26ea15144933497a98a05f83b05e8f93
02:04:07.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.179996 32706 raft_consensus.cc:2804] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 10 FOLLOWER]: Leader pre-election won for term 11
02:04:07.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.180081 32706 raft_consensus.cc:493] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 10 FOLLOWER]: Starting leader election (detected failure of leader 26ea15144933497a98a05f83b05e8f93)
02:04:07.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.180111 32706 raft_consensus.cc:3060] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 10 FOLLOWER]: Advancing to term 11
02:04:07.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.180878 32706 raft_consensus.cc:515] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:04:07.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.181207 32706 leader_election.cc:290] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [CANDIDATE]: Term 11 election: Requested vote from peers 4e39ec54c2d642feafa81dc5d4f066c2 (127.29.165.130:35181), 26ea15144933497a98a05f83b05e8f93 (127.29.165.131:44535)
02:04:07.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.181552 32647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "830b8d1af48f4d34b281a548f1dbe5ef" candidate_uuid: "06f0647651a749f8a533afd451fe5fcb" candidate_term: 11 candidate_status { last_received { term: 10 index: 40 } } ignore_live_leader: false dest_uuid: "4e39ec54c2d642feafa81dc5d4f066c2"
02:04:07.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.181677 32647 raft_consensus.cc:3060] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 9 FOLLOWER]: Advancing to term 11
02:04:07.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:04:07.182286 32399 leader_election.cc:336] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 26ea15144933497a98a05f83b05e8f93 (127.29.165.131:44535): Network error: Client connection negotiation failed: client connection to 127.29.165.131:44535: connect: Connection refused (error 111)
02:04:07.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.183581 32647 raft_consensus.cc:2468] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06f0647651a749f8a533afd451fe5fcb in term 11.
02:04:07.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.183920 32399 leader_election.cc:304] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06f0647651a749f8a533afd451fe5fcb, 4e39ec54c2d642feafa81dc5d4f066c2; no voters: 26ea15144933497a98a05f83b05e8f93
02:04:07.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.184161 32706 raft_consensus.cc:2804] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 11 FOLLOWER]: Leader election won for term 11
02:04:07.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.184358 32706 raft_consensus.cc:697] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [term 11 LEADER]: Becoming Leader. State: Replica: 06f0647651a749f8a533afd451fe5fcb, State: Running, Role: LEADER
02:04:07.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.184669 32706 consensus_queue.cc:237] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 10.40, Last appended by leader: 40, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } }
02:04:07.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.187170 30542 catalog_manager.cc:5696] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb reported cstate change: term changed from 10 to 11, leader changed from 26ea15144933497a98a05f83b05e8f93 (127.29.165.131) to 06f0647651a749f8a533afd451fe5fcb (127.29.165.129). New cstate: current_term: 11 leader_uuid: "06f0647651a749f8a533afd451fe5fcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06f0647651a749f8a533afd451fe5fcb" member_type: VOTER last_known_addr { host: "127.29.165.129" port: 45737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26ea15144933497a98a05f83b05e8f93" member_type: VOTER last_known_addr { host: "127.29.165.131" port: 44535 } health_report { overall_health: UNKNOWN } } }
02:04:07.593 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 830b8d1af48f4d34b281a548f1dbe5ef for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956' with partition [<start>, <end>)
02:04:07.594 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=830b8d1af48f4d34b281a548f1dbe5ef@[06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)[L],26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535),4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)]}]
02:04:07.594 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=830b8d1af48f4d34b281a548f1dbe5ef, attempt=12, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=1721), Traces: [0ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [0ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: connection closed, [0ms] delaying RPC due to: Network error: connection closed, [18ms] refreshing cache from master, [18ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [20ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [21ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [22ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [22ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [40ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [41ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [43ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [43ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [78ms] refreshing cache from master, [78ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [80ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [81ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [83ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [83ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [158ms] refreshing cache from master, [158ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [160ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [163ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [164ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [164ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [218ms] refreshing cache from master, [218ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [220ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, [221ms] sending RPC to server 26ea15144933497a98a05f83b05e8f93, [223ms] received response from server 26ea15144933497a98a05f83b05e8f93: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [223ms] delaying RPC due to: Network error: Failed to connect to peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535): Connection refused: /127.29.165.131:44535, [1718ms] refreshing cache from master, [1718ms] Sub RPC GetTableLocations: sending RPC to server master-127.29.165.188:38583, [1720ms] Sub RPC GetTableLocations: received response from server master-127.29.165.188:38583: OK, deferred=Deferred@1600959951(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$233/0x00007fa4f8336040@5703139b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@728615888 after retry RPC@424254333) -> (continuation of Deferred@639368020 after retry RPC@1943805593) -> (continuation of Deferred@1849135097 after retry RPC@2106354800) -> (continuation of Deferred@1478225759 after retry RPC@1189799872) -> (continuation of Deferred@1859907067 after retry RPC@2132726535), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@40ce4b13 -> wakeup thread Time-limited test -> (continuation of Deferred@728615888 after retry RPC@424254333) -> (continuation of Deferred@639368020 after retry RPC@1943805593) -> (continuation of Deferred@1849135097 after retry RPC@2106354800) -> (continuation of Deferred@1478225759 after retry RPC@1189799872) -> (continuation of Deferred@1859907067 after retry RPC@2132726535))) after lookup
02:04:07.595 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] connecting to peer
02:04:07.595 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] Successfully connected to peer
02:04:07.609 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x7169e736, L:/127.0.0.1:40154 - R:/127.29.165.129:45737] using TOKEN/null
02:04:07.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.613729 32647 raft_consensus.cc:1275] T 830b8d1af48f4d34b281a548f1dbe5ef P 4e39ec54c2d642feafa81dc5d4f066c2 [term 11 FOLLOWER]: Refusing update from remote peer 06f0647651a749f8a533afd451fe5fcb: Log matching property violated. Preceding OpId in replica: term: 9 index: 36. Preceding OpId from leader: term: 11 index: 42. (index mismatch)
02:04:07.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:04:07.614022 32399 consensus_peers.cc:597] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb -> Peer 26ea15144933497a98a05f83b05e8f93 (127.29.165.131:44535): Couldn't send request to peer 26ea15144933497a98a05f83b05e8f93. Status: Network error: Client connection negotiation failed: client connection to 127.29.165.131:44535: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:04:07.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.614408 32706 consensus_queue.cc:1048] T 830b8d1af48f4d34b281a548f1dbe5ef P 06f0647651a749f8a533afd451fe5fcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e39ec54c2d642feafa81dc5d4f066c2" member_type: VOTER last_known_addr { host: "127.29.165.130" port: 35181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 36, Time since last communication: 0.000s
02:04:07.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.620585 32724 mvcc.cc:204] Tried to move back new op lower bound from 7287070095818051584 to 7287070094070890496. Current Snapshot: MvccSnapshot[applied={T|T < 7287070095818051584}]
02:04:07.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.622615 32726 mvcc.cc:204] Tried to move back new op lower bound from 7287070095818051584 to 7287070094070890496. Current Snapshot: MvccSnapshot[applied={T|T < 7287070095818051584}]
02:04:07.634 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.29.165.131:44535
02:04:07.634 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

02:04:07.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.635301 30358 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskxPorqF/build/debug/bin/kudu
02:04:07.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskxPorqF/build/debug/bin/kudu
02:04:07.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/wal
02:04:07.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/data
02:04:07.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:04:07.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:04:07.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:04:07.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:04:07.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:04:07.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:04:07.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/logs
02:04:07.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/data/info.pb
02:04:07.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:04:07.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:04:07.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:04:07.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:04:07.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:04:07.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:04:07.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:04:07.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:04:07.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:04:07.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:04:07.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.165.131:44535
02:04:07.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.165.131
02:04:07.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.165.190:39217,127.29.165.189:39733,127.29.165.188:38583
02:04:07.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45365
02:04:07.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.165.131
02:04:07.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.165.148:38003
02:04:07.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:04:07.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:04:07.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:04:07.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:04:07.748746 32733 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:04:07.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:04:07.749042 32733 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:04:07.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:04:07.749118 32733 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:04:07.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:04:07.752727 32733 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:04:07.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:04:07.752878 32733 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.165.131
02:04:07.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.756829 32733 tablet_server_runner.cc:78] Tablet server non-default flags:
02:04:07.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:04:07.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.29.165.148:38003
02:04:07.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:04:07.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:04:07.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/data
02:04:07.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/wal
02:04:07.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:04:07.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:04:07.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.29.165.131:44535
02:04:07.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:04:07.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:04:07.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:04:07.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/data/info.pb
02:04:07.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.29.165.131
02:04:07.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45365
02:04:07.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.29.165.190:39217,127.29.165.189:39733,127.29.165.188:38583
02:04:07.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:04:07.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --heap_profile_path=/tmp/kudu.32733
02:04:07.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:04:07.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:04:07.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:04:07.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:04:07.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.29.165.131
02:04:07.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/logs
02:04:07.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:04:07.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:04:07.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:04:07.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:04:07.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:04:07.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
02:04:07.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type DEBUG
02:04:07.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 01:43:15 UTC on bdcb31816ec0
02:04:07.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11948
02:04:07.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.758235 32733 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:04:07.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.759445 32733 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:04:07.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:04:07.765738 32739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:04:07.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:04:07.765769 32738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:04:07.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 02:04:07.765833 32741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:04:07.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.766040 32733 server_base.cc:1061] running on GCE node
02:04:07.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.766505 32733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:04:07.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.767199 32733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:04:07.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.768486 32733 hybrid_clock.cc:648] HybridClock initialized: now 1779069847768465 us; error 33 us; skew 500 ppm
02:04:07.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.770483 32733 webserver.cc:533] Webserver started at http://127.29.165.131:45365/ using document root <none> and password file <none>
02:04:07.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.771117 32733 fs_manager.cc:362] Metadata directory not provided
02:04:07.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.771203 32733 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:04:07.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.774945 32733 fs_manager.cc:714] Time spent opening directory manager: real 0.002s	user 0.003s	sys 0.000s
02:04:07.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.776889 32747 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:04:07.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.777877 32733 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
02:04:07.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.778035 32733 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/data,/tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/wal
02:04:07.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "26ea15144933497a98a05f83b05e8f93"
02:04:07.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 02:03:36 on dist-test-slave-838s"
02:04:07.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.778487 32733 fs_report.cc:389] FS layout report
02:04:07.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:04:07.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/wal
02:04:07.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/wal
02:04:07.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/data/data
02:04:07.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:04:07.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:04:07.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:04:07.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:04:07.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:04:07.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:04:07.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:04:07.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:04:07.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:04:07.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:04:07.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:04:07.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.791277 32733 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:04:07.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.792037 32733 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:04:07.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.792225 32733 kserver.cc:163] Server-wide thread pool size limit: 3276
02:04:07.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.792809 32733 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:04:07.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.794130 32754 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
02:04:07.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.797344 32733 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:04:07.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.797421 32733 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.003s	user 0.000s	sys 0.000s
02:04:07.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.797502 32733 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:04:07.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.799021 32733 ts_tablet_manager.cc:616] Registered 1 tablets
02:04:07.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.799105 32733 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s	user 0.001s	sys 0.000s
02:04:07.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.799355 32754 tablet_bootstrap.cc:492] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93: Bootstrap starting.
02:04:07.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.809873 32733 rpc_server.cc:307] RPC server started. Bound to: 127.29.165.131:44535
02:04:07.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.809921   393 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.165.131:44535 every 8 connection(s)
02:04:07.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.810845 32733 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskxPorqF/test-tmp/mini-kudu-cluster5935334487790547732/ts-2/data/info.pb
02:04:07.812 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:04:07.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.812476 30358 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskxPorqF/build/debug/bin/kudu as pid 32733
02:04:07.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.818825 32754 log.cc:826] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93: Log is configured to *not* fsync() on all Append() calls
02:04:07.821 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956, tablet=830b8d1af48f4d34b281a548f1dbe5ef, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:04:07.832 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779069816956, tablet=830b8d1af48f4d34b281a548f1dbe5ef, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:04:07.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.836934   394 heartbeater.cc:344] Connected to a master server at 127.29.165.190:39217
02:04:07.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.837229   394 heartbeater.cc:461] Registering TS with master...
02:04:07.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.837815   394 heartbeater.cc:507] Master 127.29.165.190:39217 requested a full tablet report, sending...
02:04:07.838 [DEBUG - main] (Connection.java:575) [peer master-127.29.165.190:39217(127.29.165.190:39217)] disconnecting while in state READY
02:04:07.839 [DEBUG - main] (Connection.java:575) [peer 4e39ec54c2d642feafa81dc5d4f066c2(127.29.165.130:35181)] disconnecting while in state TERMINATED
02:04:07.839 [DEBUG - main] (Connection.java:575) [peer 26ea15144933497a98a05f83b05e8f93(127.29.165.131:44535)] disconnecting while in state TERMINATED
02:04:07.839 [DEBUG - main] (Connection.java:575) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] disconnecting while in state READY
02:04:07.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.840302 30400 ts_manager.cc:194] Re-registered known tserver with Master: 26ea15144933497a98a05f83b05e8f93 (127.29.165.131:44535)
02:04:07.840 [DEBUG - main] (Connection.java:575) [peer master-127.29.165.189:39733(127.29.165.189:39733)] disconnecting while in state READY
02:04:07.840 [DEBUG - main] (Connection.java:575) [peer master-127.29.165.188:38583(127.29.165.188:38583)] disconnecting while in state READY
02:04:07.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.842624   395 heartbeater.cc:344] Connected to a master server at 127.29.165.188:38583
02:04:07.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.842741   395 heartbeater.cc:461] Registering TS with master...
02:04:07.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.842949   395 heartbeater.cc:507] Master 127.29.165.188:38583 requested a full tablet report, sending...
02:04:07.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.843688 30543 ts_manager.cc:194] Re-registered known tserver with Master: 26ea15144933497a98a05f83b05e8f93 (127.29.165.131:44535)
02:04:07.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.844591 30543 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.29.165.131:33113
02:04:07.844 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] handling channelInactive
02:04:07.845 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 06f0647651a749f8a533afd451fe5fcb(127.29.165.129:45737)] cleaning up while in state READY due to: connection closed
02:04:07.845 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.29.165.188:38583(127.29.165.188:38583)] handling channelInactive
02:04:07.845 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.29.165.188:38583(127.29.165.188:38583)] cleaning up while in state READY due to: connection closed
02:04:07.845 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2704) Releasing all remaining resources
02:04:07.846 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.29.165.189:39733(127.29.165.189:39733)] handling channelInactive
02:04:07.846 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.29.165.189:39733(127.29.165.189:39733)] cleaning up while in state READY due to: connection closed
02:04:07.846 [DEBUG - kudu-nio-2] (Connection.java:239) [peer master-127.29.165.190:39217(127.29.165.190:39217)] handling channelInactive
02:04:07.846 [DEBUG - kudu-nio-2] (Connection.java:709) [peer master-127.29.165.190:39217(127.29.165.190:39217)] cleaning up while in state READY due to: connection closed
02:04:07.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.847810 30358 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskxPorqF/build/debug/bin/kudu with pid 32382
02:04:07.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.851776   396 heartbeater.cc:344] Connected to a master server at 127.29.165.189:39733
02:04:07.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.851850   396 heartbeater.cc:461] Registering TS with master...
02:04:07.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.852017   396 heartbeater.cc:507] Master 127.29.165.189:39733 requested a full tablet report, sending...
02:04:07.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.852870 30471 ts_manager.cc:194] Re-registered known tserver with Master: 26ea15144933497a98a05f83b05e8f93 (127.29.165.131:44535)
02:04:07.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.861393 30358 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskxPorqF/build/debug/bin/kudu with pid 32564
02:04:07.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.868261 30358 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskxPorqF/build/debug/bin/kudu with pid 32733
02:04:07.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.868290 32754 tablet_bootstrap.cc:492] T 830b8d1af48f4d34b281a548f1dbe5ef P 26ea15144933497a98a05f83b05e8f93: Bootstrap replayed 1/1 log segments. Stats: ops{read=40 overwritten=0 applied=40 ignored=0} inserts{seen=30 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:04:07.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.874296 30358 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskxPorqF/build/debug/bin/kudu with pid 30369
02:04:07.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.880513 30358 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskxPorqF/build/debug/bin/kudu with pid 30438
02:04:07.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 02:04:07.886073 30358 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskxPorqF/build/debug/bin/kudu with pid 30510
02:04:07.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-05-18T02:04:07Z chronyd exiting

Time: 33.695

OK (1 test)

